CRESST REPORT 760 



Noelle C. Grijfin THIRD YEAR REPORT: 
Judy N. Miyoshi EVALUATION OF THE 
ARTFUL LEARNING 
PROGRAM 

AUGUST, 2009 




National Center for Research on Evaluation, Standards, and Student Testing 

Graduate School of Education & Information Studies 
UCLA University of California, Los Angeles 




11 




Third Year Report: Evaluation of the 
Artful Learning Program 



CRESST Report 760 



Noelle C. Griffin, and Judy N. Miyoshi 
CRESST/University of California, Eos Angeles 



July, 2009 



National Center for Research on Evaluation, 
Standards, and Student Testing (CRESST) 
Center for the Study of Evaluation (CSE) 
Graduate School of Education & Information Studies 
University of California, Eos Angeles 
300 Charles E. Young Drive North 
GSE&IS Building, Box 951522 
Eos Angeles, CA 90095-1522 
(310) 206-1532 




Copyright © 2009 The Regents of the University of California 

The work reported herein was supported under a grant from the Grammy Foundation. The findings 
and opinions expressed in this report do not reflect the positions or policies of the Grammy 
Foundation. 




THIRD YEAR REPORT: 



EVALUATION OF THE ARTFUL LEARNING PROGRAM 

Noelle Griffin, & Judy N. Miyoshi 
CRESST/University of California, Los Angeles 

Abstract 

The National Center for Research on Evaluation, Standards, and Student Testing 
(CRESST) at UCLA was contracted to undertake a three-year external evaluation of the 
Artful Learning program, an arts-based school improvement model developed from the 
work and philosophy of the late composer Leonard Bernstein. This is the third-year 
report of evaluation findings, with a primary focus on Artful Learning participants in the 
2003-2004 school year. The purpose of this report is to provide information about the 
implementation and impact of the program at current participating school sites, as well as 
place these findings within the context of the overall findings from the three-year 
evaluation as a whole. Multiple quantitative and qualitative data collection methods were 
employed throughout this evaluation. Overall, the findings suggest that the Artful 
Learning program was a useful tool for teachers with a variety of previous teaching 
experience, district and state contextual demands, grade/content areas taught, and student 
populations. Teacher satisfaction with the professional development components of the 
program were high, although assessment was an area singled out as needing additional 
support. Recommendations, drawing from all three years of the evaluation, are also 
discussed. 



Introduction/Background 

The Artful Learning/Leonard Bernstein Center Model is an arts-based sehool 
improvement model developed from the work and philosophy of the late eomposer Leonard 
Bernstein. The model includes both school- and classroom-level components. At the 
classroom level, the focus is on teachers using art as an entree to all aspects of the 
curriculum, from language arts to math, science, and history. At the school level, there is an 
emphasis on collaboration and shared leadership. 

The Artful Learning model was developed by the Leonard Bernstein Center in 1992. 
The model has been sponsored by the Grammy Foundation since 1999. The New American 
Schools endorsed the program as a comprehensive school reform model after an extensive 
review of achievement results and a site visit. A goal of the Grammy Foundation is inclusion 
of the model in the North West Regional Education Lab’s (NWREL) list of approved school 
reform models. Over the years, the model has been used in over 40 schools across the 
country. At the time of this report’s writing, 21 schools are actively using the model. 
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The Artful Learning eurrieulum is implemented through Artful Learning units, or 
instruetional modules organized around a speeific Masterwork of art, along with an 
underlying concept (i.e., a theme that the Masterwork will help express), and a significant 
question to be explored throughout the unit. The Masterwork eoncept and signifieant 
question are used as starting points and organizing constructs for the instructional unit. The 
term Masterwork within the Artful Learning units often refers to what is traditionally 
considered the arts, such as a musical piece, painting, or literature, but can also be used to 
refer to any endeavor or expression of cultural significance. Examples of recent masterworks 
used in Artful Learning classrooms include a range of events, people, and structures: The 
Gold Rush, Woody Guthrie songs, Alexander Calder sculptors, the play A Raisin in the Sun, 
drawings by graphic artist M. C. Escher, and the Bay and Brooklyn Bridges. 

Each Artful Learning unit is comprised of an instructional process with four 
interlocking phases: Experience (the students interact with the masterwork). Inquire (they 
initiate investigation and research triggered by the masterwork). Create (they design their 
own original creation), and Reflect (they use various tools to deepen their understanding of 
what they have learned and their process of learning it). The Artful Learning model extends 
beyond the classroom to promote school-level structures/organization that will best support 
curricular reform. There is an emphasis on shared leadership and collaboration among school 
staff and stakeholders. Additionally, the Artful Learning model seeks to develop a vision of 
the arts as both a school-level priority and an integral part of the educational process. 

In addition to the curricular model itself, the Grammy Foundation also provides support 
to each participating school for program implementation. The foundation offers 5 days of 
initial professional development at each school (Eevel 1 training), as well as designed 
scaffolded follow-up training for returning schools (Eevel 2 & 3 training). There are 
principals’ institutes offered during the year to actively involve administrators in the 
implementation and school change process. Each school is also encouraged to designate an 
individual to serve as the Artful Eearning coach/facilitator who is the school’s on-site 
consultant during the Artful Eearning implementation process. 

The National Center for Research on Evaluation, Standards, and Student Testing 
(CRESST) at UCLA was contracted by the Grammy Foundation to undertake a 3-year 
external evaluation of the Artful Eearning program. This is the third-year report of evaluation 
findings, with a primary focus on Artful Teaming participants in the 2003-2004 school year. 
Over the course of the 3 -year evaluation study, a number of school sites have joined the 
Artful Eearning program and a few school sites have ceased formal implementation of the 
program. 
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The changes in the specific sites involved in the program from year-to-year, in addition 
to staff turnover at individual schools, makes year-to-year comparisons of schools difficult. 
Although we did not follow “dropout” school sites after they had decided to leave the 
program, the majority of these sites appear to have made this decision based on changes in 
leadership at the school site or at the district level, or due to district curricula mandates and 
demands that precluded the fair implementation of the Artful Learning program. The purpose 
of this report, however, is to provide information about the implementation and impact of the 
program at currently participating Artful Learning school sites, including returning school 
sites and those schools new to the program. Before presenting our results, we briefly describe 
the methodology used for this report. At the end of the report, we provide some 
recommendations for both the schools and program designers based on these findings. 

Methodology 

The evaluation findings presented in this report are based on the integration of several 
types of quantitative and qualitative data. These data sources include: interviews with 
administrators and teachers, pre- and post-professional development surveys, school-level 
implementation surveys, and preliminary school-level achievement data. 

Teacher Interviews 

CRESST researchers interviewed a sample of teachers from participating Artful 
Learning schools in Winter 2003 through Spring 2004. The majority of these interviews were 
conducted by telephone, although a few at Los Angeles-area schools were conducted in 
person. The goal was to interview 2 teachers from each of the 21 currently participating 
Artful Learning schools. However, several of the newer-implementing schools had yet to 
have teachers fully use the Artful Learning program in the classroom, and thus were unable 
to have teachers participate in the interviews. Ultimately, a total of 24 teachers from 14 
schools participated in the interviews. The teachers ranged in previous teaching experience 
from first year to over 40 years teaching experience. The majority of teachers (16) taught at 
the primary grades, with 8 teaching at the secondary level. For 10 of the teachers, this was 
the first year implementing the Artful Learning program; the remainder of teachers ranged 
from 1 to 6 previous years of Artful Learning program usage. 

Researchers used a semi-structured protocol for the interviews. The protocol focused on 
a number of areas, including teachers’ understanding of the Artful Learning model, their 
experiences in implementing the model’s components, impact of the model (on students, 
teachers, and the school as a whole), and successes and roadblocks in the implementation 
process. The interviews also gathered information about teacher background, experiences. 
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and general instructional practices. Interviews were audio taped, transcribed, and finally 
analyzed using Atlasti qualitative data analysis software. A sample teacher interview protocol 
is included in Appendix A. 

Administrator Interviews 

Researchers conducted one-on-one telephone interviews with administrators of Artful 
Learning schools from Winter 2003 through Spring 2004. The administrators were from 
schools at all phases of the Artful Learning implementation process, including both newly 
implementing schools and schools that had been implementing the program for several years. 
Administrators at all 21 of the current Artful Learning participant schools, both new and 
returning, were invited to participate in interviews, and a total of 1 5 administrators from 9 of 
these schools agreed. 

The purposes of these semi-structured interviews were both to collect information about 
the quality of school-level implementation of the various facets of the Artful Learning model 
and contextual factors surrounding that implementation. Specific successes and potential 
roadblocks that were encountered in this implementation process were also addressed. 
Transcribed interviews were analyzed using Atlasti. A sample interview protocol can be found 
in Appendix A. 

Professional Development Surveys 

All teachers attending Grammy-offered Artful Learning professional development 
sessions during the 2003-2004 school year were given two surveys, one at the begirming of 
the training and one after training completion. These surveys focused on both teachers’ 
perceptions of the quality of the professional development and their own level of expertise on 
the topics covered during the training sessions. The pre- and post- training surveys are in 
Appendix A. 

A total of 468 teachers completed at least some component of the professional 
development surveys. 199 teachers provided complete pre- and post-training survey sets, 112 
from the Level 1 training, and 87 from the Level 2 training. The professional development 
survey results were analyzed using SPSS 12.0, a statistical software package. Both basic 
descriptive and inferential statistical analyses are presented in this report. 

Beyond issues of institute quality and utility, the surveys also asked teachers a series of 
questions set to gauge changes in expertise due to their professional development 
participation. These items required teachers to rate their expertise on a variety of topics 
covered during the professional development both at the beginning and end of the session. 
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The individual items were used to develop four scales each from the pre- and post- 
professional development surveys. Each scale focused on a key piece of the Artful Learning 
professional development content: Artful Learning Process (the unique classroom instruction 
components of the model), Assessment (the model’s assessment strategies), Organization 
(the school-based components of the model), and General Instructional Practice (aspects of 
quality instructional practice that are linked with — ^but not unique to — the Artful Learning 
model). Sample items for each of the scales can be found in Table 1. Table 1 also presents a 
measure of scale reliability (coefficient alpha) for each of the pre- and post-professional 
development scales for both the Level 1 and the Level 2 training, as item content varied 
slightly for Level 1 and Level 2. As this table shows, all of the scales have acceptable 
measures on internal consistency. 



Table 1 

Professional Development Expertise Scales: Sample Items and Reliability 







Coefficient Alpha 


Scale 


Sample Items 


Pre 


Post 


Artful Learning 
Process 


Selecting appropriate works of art for use 
in my curriculum. 

Developing questions for students to use in 
conducting inquiry/research as part of their 
regular classroom activities. 


Level 1 = .92 
Level 2 = .96 


Level 1 = .90 
Level 2 = .95 


Assessment 


Using a portfolio assessment system. 

Using student assessment results to plan 
and refine my classroom practices. 


Level 1 = .92 
Level 2 = .91 


Level 1 = .92 
Level 2 = .92 


Organization 


Conducting action research at the school 
level to examine school-wide curriculum 
and instruction. 

Coordinating my curriculum with that of 
other teachers at my school. 


Level 1 = .88 
Level 2 = .90 


Level 1 = .90 
Level 2 = .90 


General 

Instructional 

Practices 


Coordinating my curriculum with district 
standards. 

Coordinating my curriculum with national 
standards 


Level 1 = .86 
Level 2 = .87 


Level 1 = .86 
Level 2 = .86 



School-level Implementation Survey 

In Winter/Spring 2004, teachers at all participating Artful Learning program schools 
were asked to complete an implementation survey. Survey items focused on a number of 
factors, including: implementation of Artful Learning model processes and components, 
overall instructional and curricular practices, assessment, and organizational/school-level 
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characteristics. CRESST received 390 eompleted teacher surveys from 16 sehools. Response 
rates by sehools ranged from 10% to 94%, with an average response rate across the sehools 
of 53%. 

The implementation survey is in Appendix A. The survey items were used to ereate a 
number of seales addressing both elassroom practice and sehool-level faetors that play 
important roles in the Artful Learning model and its underlying theory. Additionally, a seale 
was designed to focus specifically on the unique arts-based elassroom proeesses of the Artful 
Learning model. Table 2 presents each of these scales along with a statistieal measure of 
internal consisteney. Coefficient Alpha. Note that the eoefficient alphas for all of the seales 
are in the range generally agreed upon as aeeeptable in the soeial seienees (Netemeyer, 
Bearden & Sharma, 2003). 



Table 2 

Artful Learning Implementation Survey Scales 



Scale 


Content 


Coefficient Alpha 


Assessment Use/Understanding 


Use of multiple assessment 
strategies/understanding of how to use 
assessment 


.74 


Standards Use/Understanding 


Use of standards in planning and instruction 


.77 


Quality Instructional Practices 


Quality of overall instructional techniques 


.78 


Parent Involvement 


Implementation of activities designed to foster 
parent participation in school activities and 
understanding of school processes 


.70 


Shared Mission 


Perception of an overall shared instructional 
mission/vision for the school 


.89 


Shared Leadership 


Participation of multiple stakeholders in school 
decision-making and leadership processes 


.80 


Artful Learning Process 
Implementation 


Quality of implementation of Artful Learning 
instructional processes 


.85 


Reported Impact 


Extent of program impact on student 
engagement, learning, and achievement 


.90 



All implementation surveys were analyzed using SPSS 12.0 statistieal software. Both 
basie descriptive statisties and inferential analyses are presented in this report. 

Preliminary School-level Achievement Data 

The schools included in this report represent a variety of levels of Artful Learning 
implementation. Lor measurable student achievement outeomes to be assessed, there needs to 
be both an extended period of program implementation and several years of data available. 
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which is only the case for a small subset of current Artful Learning participants. At present, 
the only sourees CRESST has for these sehools’ achievement information is publiely 
available school-level data provided on state and/or distriet websites. There are several 
limitations to this type of data, including: 

The school-level nature of the data. The laek of individual student- and teacher-level 
data makes it very diffieult to identify achievement change and attribute it to Artful Learning 
program participation, particularly as many of the schools have some teaehers who have yet 
to implement the program in their classrooms. Any Artful Learning aehievement impact at 
the sehool-level may be diluted by teachers who have yet to receive the training or use the 
program, and there is no way to separate these teaehers out of the analysis in the current data 
sets available. Lurthermore, the data presented on most of these sehool databases is eross- 
seetional — that is, presenting data for a given grade from year-to-year. This approaeh is 
limited in that any ehanges ean be attributed to eohort effeets (i.e., differences between the 
groups of students enrolled at a school from year-to-year) as opposed to program 
implementation. 

Test comparison problems. The Artful Learning sehools use a variety of state or 
distriet-based tests in reporting their sehool-level data, as well as reporting their data in 
different formats. This makes aggregation for running inferential statistics (i.e., looking at the 
overall ehanges in aehievement for Artful Learning sehools eombined) very difficult. 
Similarly, many of the schools and districts involved have switehed achievement tests onee 
or more sinee joining the Artful Learning program. Again, this interferes with the ability to 
monitor ehange over time, as there is not a eonstant aehievement measure in plaee. 

With these caveats in mind, we present in this report a tentative deseription of sehool 
level aehievement data. This reporting foeuses on the pereentage of students meeting or 
exceeding state standards based on eaeh of the schools’ states’ assessment systems. We 
ineluded information for nine Artful Learning sehools that have both been involved in the 
program for at least 3 years and have appropriate retrospective aehievement data available on 
their state/distriet websites. We also ineluded, for eomparison purposes, similar information 
about the distriets as a whole that these sehools reside in, and about a set of matehed 
comparison schools. A total of two matched eomparison sehools were seleeted for each of 
the 9 Artful Learning sehools, from the same districts as the Artful Learning sehools, 
resulting in 18 eomparison schools total. These eomparison schools were selected by 
matching them to the Artful Learning schools based on their similarity aeross a number of 
faetors, including: school size, teacher/student ratio, pereentage of English Language 
Learners, percentage of free/redueed luneh students, and student demographies. The deeision 
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was made to focus on information for Grades 4 and 8, as these were the two most 
consistently tracked grades over time across the schools. 

As student-level data was not available for the Artful Learning schools or their 
comparisons, it is thus difficult to ascribe any differences in achievement growth specifically 
to Artful Learning participation. The lack of student-level data, in conjunction with the small 
sample size, also do not allow for the running of inferential statistical tests to determine 
statistical significance. What the data does provide is some descriptive comparisons between 
the Artful Learning and comparison schools in the changes over time of the percentage of 
students meeting/exceeding state standards, and some suggestions of areas that would be 
fruitful ground for additional analyses with student level achievement data. 

Findings 

The following findings represent an integration of quantitative and qualitative data. 
First, we present findings about the success and quality of Artful Learning program 
implementation at the participating sites, including analysis of the overall implementation 
process and the implementation of the individual instructional components of the model. 
Issues relating to cross-content curriculum integration and assessment are also discussed. 
Second, we address findings related to program impact, including professional development 
impact, school- and classroom-level predicators of implementation, teacher-reported student 
impact, and student achievement comparisons. Finally, we discuss potential facilitators and 
barriers to the Artful Learning program’s implementation and persistence of use, based on 
the experiences of the participating schools. 

Artful Learning Implementation 

The Artful Learning model has multiple layers to it, and requires teacher time, effort, 
and creativity for full implementation. Rather than an add-on program that can be “dropped 
into’’ a school, implementation of Artful Learning is more of an evolving process, with on- 
going development building on previous implementation and successes. Thus, 
implementation is a critical component of the overall success of the Artful Learning model. 
This section of the report discusses findings regarding the implementation of the model as a 
whole and its various components. 

Overall implementation process. An important initial question in addressing the 
implementation of the Artful Learning program is whether, in fact, teachers are using the 
program as part of their classroom instruction. The surveyed teachers responded to several 
questions related to their general use of the program. Overall, their responses suggest that the 
majority of teachers who go through the Artful Learning training continue to use the program 




in the classroom after initial implementation. For example as shown in Figure 1, all but 8% 
of the responding teachers reported that they had developed an Artful Learning unit during 
the current school year, and almost half (41%) of the teachers had designed 3 units or more. 




Figure 1. Teacher survey responses: Approximately how many Artful 
Learning Units have you planned this school year? {N= 375) 



Beyond the Artful Learning units as a whole, the teachers were asked how often they 
actually incorporated the arts into their instruction. Again, the majority of teachers, over 50% 
reported using the creation of art in their instruction “Often,” “Very Often,” or “Always,” 
with only 2% stating that they never used works of art as part of their regular classroom 
instruction. Furthermore, approximately three quarters of the teachers responded similarly 
that they explicitly identified the types of artistic experiences (visual, kinesthetic, auditory) 
students will engage in as part of their unit planning. These results are presented in Figures 2 
and 3, respectively. 




Figure 2. Teacher survey responses: How often do you incorporate the creation 
of an artistic work into your instruction? (N= 369) 
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Figure 3. Teacher survey responses: When planning an Artful Learning Unit 
do you explicitly identify the visual, kinesthetic, or auditory experiences your 
students will engage in? (A= 359) 



Outside of their individual elassrooms, teachers’ perceptions of their schools’ use of the 
arts as a whole was consistent with their individual experiences. Specifically, as displayed in 
Figure 4, over 70% of the responding teachers “agreed” or “strongly agreed” that arts are 
incorporated into regular instruction at their schools. 




Figure 4. Teacher survey responses: At my school, arts instruction is 
incorporated into instruction in other subject areas. (N= 370) 



The implementation level at the schools already using the Artful Learning program 
appears to be growing as well, based on the teacher survey responses. For example, almost 
60% of the teachers reported that the number of teachers using the program at their school 
and their own use of the program in the classroom had increased compared to previous 
implementation years. Additionally, 66% of the teachers felt that their understanding of the 
Artful Learning program had continued to increase. These findings are presented in Figures 
5, 6, and 7. 
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Figure 5. Teacher survey responses: Changes in Artful Learning 
implementation this year compared to previous years. (N= 333) 




Figure 6. Teacher survey responses: Changes in Artful Learning classroom 
use this year compared to previous years. {N= 334). All percentages have 
been rounded up to the nearest whole number. 




Figure 7. Teacher survey responses: Changes in understanding of Artful 
Learning components this year compared to previous years. (N= 332). 
All percentages have been rounded up to the nearest whole number. 
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The professional development post-test survey responses of Artful Learning veteran 
teachers (i.e., teachers with at least 1 year prior experience using the program in the 
classroom) provide additional evidence about teachers on-going implementation/satisfaction 
with the program. For example, 83% of these teachers “agreed” or “strongly agreed” that 
they were looking forward to continued use of the Artful Learning program in their 
classrooms, and responded similarly when asked whether the program is useful to them in 
their day-to-day classroom practices. Additionally, 90% of these teachers “agreed” or 
“strongly agreed” that there was a need for the Artful Learning program at their school. 
These results are presented in Figures 8, 9, and 10, respectively. 



100%n 



80 %H 




Strongly Disagree Disagree Agree Agree Strongly 

disagree slightly slightly agree 

more than more than 
agree disagree 



Figure 8. Teacher responses: I am looking forward to continuing to use the 
artful learning program. (N= 135) 
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Figure 9. Teacher responses: The artful learning program is useful in my day- 
to-day practice. (N= 139) 
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Figure 10. Teacher responses: There is a need for artful learning at my school. 

(A^=139) 

Analysis of qualitative data supports these survey findings. Overall, both new and 
Artful Leaming-experieneed teaehers reported that they approached the planning and 
implementation of Artful Learning units with great anticipation and excitement. Although 
some concerns emerged for new teachers, many of them reported that they were encouraged 
by how the program is working for their students and remained realistic about the program. 
That is, several of the teachers acknowledged that it will take some time to perfect their 
Artful Learning-based units. For example: 

Even when we were asked to give up 4 summer days for our training last year everyone 
left that training very energized and excited about what we had been shown, so I think 
the general consensus would be a very positive outlook in the sense that it’s going to give 
our students one more thing that will help them in their learning, and in their 
development as a whole child. 

I think we’re excited about it, and we’re seeing a lot of neat things with it. I think it can 
be stressful just because it’s something new. I do think the students are reacting really 
well to it, so that has made a difference. I’d say overall that it’s positive. 

Implementation of program components. Although teachers’ overall reports about 
implementation were generally positive, there were some differences based on the 
component of the unit involved: Experience, Inquire, Create, or Reflect. When asked the 
percentage of time they spent in their classroom on each of the 4-unit component, 
comparatively less time appeared to be spent on the Reflect component than the other three 
components. For example, over 10% of teachers reported that they spent no time in their 
classrooms on the Reflect component, compared to 7%, 8%, and 5% for Experience, Create, 
and Inquire, respectively. Similarly, a smaller number of teachers (25%) reported spending 
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50% or more of their unit time on Reflect than spending 50% or more of unit time on 
Experience (38%), Create (50%), ox Inquire (44%). 

Interview findings also suggested that teaehers eontinue to experienee more diffieulty 
implementing some eomponents of the model than others. In regard to eoneeptual 
understanding and implementation of the four components — Experience, Create, Inquire and 
Reflect, new Artful Learning teachers reported that the Experience and Create components 
were the easiest to develop and implement, and that the Inquire and Reflect were the most 
challenging. The Artful Learning-experienced teachers reported that the Inquire component 
remained the most challenging for them, but that with experience and practice they had better 
mastery of the Reflect component. The Artful Learning-Experienced administrators similarly 
reported that the Experience component appeared to be the easiest for teachers to implement 
and Inquire the most challenging. 

Experience was the easiest to implement for teachers because there isn’t as much risk 
involved. It’s a low anxiety type of experiencing. It’s the beginning step of the project, I 
think. 

The most difficult is developing the Inquire Centers. That’s the hardest one. You can 
think of questions, and you can think of activities, but to come up with the materials that 
go into the Inquire Centers is the most difficult. That’s the biggest challenge, I think. 

Many teachers often cited issues concerning time as the reason why the Inquire 
component posed difficulties, but one teacher’s comment was quite revealing in regard to 
another difficulty with implementing the Inquire component. She reported that teachers felt 
challenged by the possibility of not being able to answer the questions generated by students 
as part of the Inquiry activities. 

A lot of teachers don’t want students to ask them questions they don’t know. So Inquire 
can be difficult because if you are intimidated by a child asking you something you don’t 
know the answer to or maybe asking you something that you didn’t plan for. 

In terms of teacher feedback on using other parts of the program, both new and Artful 
Learning-experienced teachers reported that they have a clear understanding of issues such as 
curricular mapping and Masterworks. They ascribed this understanding to the professional 
development they received Ifom the Grammy Foimdation. For example, interviewed teachers 
reported on ways in which curricular mapping is organized and operationalized at their school site. 

For the curricular mapping process we take our whole year’s curriculum and map it out 
week by week, indicating for each subject area what we planned to cover and the 
different resources and the different things that we needed to have in there. We needed to 
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have our significant question, our concept and the Masterwork that would be covered 
within that unit. 

Actually, it works very well because we’ve done some curricular mapping. I guess it was 
the first year we did curricular mapping. That was where the teachers met to clump 
together all of the big objectives, and the big areas that they were studying. They got all 
that work down on every grade level. They know where they needed to go. They had 
their little maps, so they knew where they needed to go. 

Additionally, both new and Artful Learning-experienced teachers reported very little 
problems with selecting a Masterwork and described some of the ways in which they are able 
to embrace the concept of Masterworks and, in turn, find ways for their students to embrace 
the selected Masterwork. 

It provides a launching off point. I know that most (Artful Learning) units can launch 
from multiple places. For me, (a masterwork) is a great place to start because you can 
take a look at an art or listen to some art of whatever the Masterwork is, and interpret it in 
a variety of ways. There are multiple ways of looking at any piece of artwork. One being 
if it’s the best that someone’s ever created or if it’s something that spans time, and it 
becomes a timeless piece of art or its intergenerational in that when people hear it today, 
it’s the same as hearing it or seeing it or experiencing it 50 years ago or 100 years ago. 
Children today sometimes don’t often have access to that information to the 
Masterworks, so, for me, it’s a great launching point for a course of study or for a path of 
thinking. Even if they don’t know it right away, they might not understand right away 
why they’re looking at it because 1 just want them to interpret it in as many ways as they 
can visually. Can they infer things from it? What do they take literally from it? 

Cross-content integration. An on-going issue teachers have reported throughout the 
last 3 years of Artful Learning evaluation has been the integration of the model across all 
content areas. Optimally, the Artful Learning process should be embedded in instruction 
across all content areas and grade levels, rather than being treated separately as “arts 
instruction.” The responses of surveyed teachers suggest that they are attempting to address 
this issue by weaving Artful Learning processes and activities throughout the curriculum. 
Figure 1 1 shows us, for example, that 88% of teachers either “agreed” or “strongly agreed” 
that they made connections to other subject areas besides art when planning Artful Learning 
units. 
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Figure 11. Teacher survey responses: When planning an Artful Learning Unit, 
I make connections to other subject areas. (N= 355) 



However, in terms of the actual content areas that teachers incorporated their Artful 
Learning instruction into, teacher’s survey responses did yield some differences. Not 
surprisingly, more teachers reported integrating Artful Learning into their social studies 
activities (63%) than into other content areas followed closely by language arts. Conversely, 
fewer teachers reported integrating Artful Learning into their math instruction (33%) than 
any of the other content areas. These results, presented in Figure 12, suggest that teachers 
have found it easier to integrate the arts into some subject areas than others, or may be better 
able to appreciate possible links in certain subject areas. For example, there may be more 
intuitive opportunities for connections between language arts and the arts than between math 
and the arts, thus making teachers less likely to attempt to integrate the arts into their math 
instruction. 
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Math Science Social Language 

Studies Arts 



Figure 12. Teacher survey responses: Subject areas incorporating an Artful 
Learning Unit. {N =390) 
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The qualitative data supports the quantitative findings, but also elieited that experienee 
with the Artful Learning program also plays a role in teachers’ confidence and ability to 
integrate the program across content areas. That is, when it comes to cross-content 
integration, many of the new teachers reported that they experienced challenges in 
mathematics instruction. The majority of the Artful Learning-experienced teachers, on the 
other hand, reported that they integrated the Artful Learning program across language arts, 
reading, mathematics, science, and social studies with very few problems. These teachers 
reported that their practice with integrating the program across content areas over the years 
has resulted in their confidence to attempt cross content integration with other units. 

New Teacher 

It was hard to do with math. One of our Inquire Centers did have a map component to it, 
but we feel like, and, again, I don’t teach math, but the math teacher talked about how 
she really had to take out 2 or 3 weeks of math in order to do one of the Inquire Centers 
that was closely related to math. That’s really hard to get away from math for 3 weeks in 
sixth grade. 

Artful Learning-experienced teacher 

They integrate quite well. I think we do a really good job teaming together in really 
trying to connect that to any other programs that we’re using. 

Assessment implementation. Student assessment remains one of the most difficult 
aspects of the Artful Learning program for teachers to address. Assessment of student 
learning not only allows teachers to track student growth, but also assists them in tailoring 
instruction to the ongoing learning needs of students. On one hand, regular assessment of the 
skills that students gain through Artful Learning involvement requires different lenses than 
what is provided by the other assessments that teachers typically have access to. On the other 
hand, heavy district and state assessment requirements often over-burden teachers, making 
the implementation of additional. Artful Learning-focused assessment materials difficult. For 
example, only 12% of the surveyed teachers reported that they had implemented student 
portfolio assessments as part of the Artful Learning program. The remainder of teachers 
either did not use portfolios at all (22%), or used a different portfolio system implemented 
prior to beginning the Artful Learning program (66%). Similarly, as show in Figure 13, only 
41% of teachers reported any increase in their use of Artful Learning assessments in the last 
year, while 12% reported that their use had decreased. 
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Figure 13. Teacher survey responses: Changes in Artful Learning Portfolio 
Assessment use this year compared to previous years. (N= 322). All percentages 
have been rounded up to the nearest whole number. 



Analysis of qualitative data provides further evidenee that, for a variety of reasons, the 
assessment component of the Artful Learning program remains a challenge for teachers to 
fully embrace and use with their students. The vast majority of teachers reported that district 
or state-mandated assessment requirements take precedence over using the Artful Learning 
assessment model. Additionally, many teachers reported that the Artful Learning assessment 
model seems to conflict with their existing portfolios systems. Because of this overlap, 
several teachers reported that they did not see the need to overhaul their current assessment 
practices, especially when it comes to student portfolios. Many teachers did, however, report 
that they like how the Artful Learning model calls upon teachers to be more reflective about 
the delivery of their instructional strategies, which in turn calls upon students to be more 
reflective about their own learning and progress in the assessment context. 

Having the students look more regularly at their progress (is a change). Having them look 
at their work on a regular basis. Having them think about how they learned looking 
through their work and their samples of work. How did they learn? How did that help 
them? How was it different than the way that we learned before? I think more of the 
student reflection part and doing it more regularly. 

I’ve always been fmstrated with my portfolios because once a trimester we sit down and 
gather up their work, and put it in, and it’s not really very reflective. I don’t think I did a 
really good job of that in this unit, but 1 can definitely see how the Bernstein (Artful 
Learning) and the reflection piece could change that. I think that’s something next year 
that I’ll do differently. Although I used portfolios, I definitely think about them 
differently, and it gives me a way to make it more meaningful. 
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Although many interviewed teaehers felt the pressure for students to perform well on 
state- and distriet-mandated assessments, they also expressed a desire for additional 
assessment support surrounding Artful Learning eontent. In fact, some of the teachers 
reported that they would like to participate in developing additional tools to help rate student 
work that is generated from the Artful Learning components. 

Implementation of Artful Learning organizational components. An important 
aspect of the Artful Learning program is its’ organizational/collaborative processes. For 
schools and teachers at the begirming levels of implementation, this component is actualized 
as shared collaboration and development of Artful Learning units, as well as increased 
collaboration between teachers from various academic departments or “specialties” (e.g., 
language arts, sciences, arts). At schools that have been implementing the Artful Learning 
program for longer periods of time, this organizational and collaborative component is 
expanded to include the use of Artful Learning cadres, or teams of school participants and 
critical friends focused on various academic and organizational support issues. Given the 
focus on continued development of school organizational and collaborative structures 
throughout the model, it is not surprising that the majority of teachers survey respondents 
reported that the amount of collaboration at their school had increased this implementation 
year compared to previous years (see Figure 14). Similarly, as shown in Figure 15, the 
majority of teachers also reported that the support that their administrators provided for the 
Artful Learning implementation as a whole had increased as well. 




Figure 14. Teacher survey responses: Changes in teacher collaboration this year 
compared to previous years. {N= 335) 
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Figure 15. Teacher survey responses: Changes in administrative support for 
Artful Learning this year compared to previous years. (A= 335) 



Data from teacher and administrator interviews provides a more in-depth picture of the 
implementation of the Artful Learning programs organizational components at participating 
school sites. Both new and Artful Learning-trained teachers and all administrators reported 
that the implementation of the Artful Learning program has resulted in the initiation of or 
increase in teacher collaboration. Several teachers reported that collaboration had become 
more inclusive as a result of the Artful Learning program, broadening to include general 
education, fine arts and special education teachers. Additionally, communication occurred 
both within and across grade levels at some school sites, which provided continuity and 
direction for all teachers. 

There is more communication between the grade level teachers that was not there before. 

There is more collaboration. We were just sitting in the lunch room yesterday, and 
another grade level were talking about the original creation that their students were 
doing, and it was just amazing to be sitting there talking about how the students were 
responding to it. Yes, I think it’s across grade levels it’s more than normal because we’re 
all talking the same language. We’re all doing the same kinds of things. 

I think it’s drawn us together, and I think it has given us a common ground to have some 
really deep conversations about learning and students achievement, and how school 
should work together in a team setting, and what collaboration should look like. So I 
think in terms of that culture, it has made a big difference especially in the role of the 
P.E. teacher, the art teacher, the music teacher, and the talent gifted teacher. It has helped 
us connect all of these. Also the Spanish teacher and the media specialist, it has helped us 
connect these people more into the core curriculum, and the day-to-day work that we do 
with teaching children all the things that we teach. So there’s been a tremendous 
connectivity part that I hadn’t anticipated. 
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Artful Learning Impact 

As described above, the Artful Learning program is an integrated approach to school 
transformation with multiple components. Thus, the program is expected to have impact in a 
variety of areas, including teacher knowledge and practice as well as short- and long-term 
student outcomes. This section of the report addresses program impact from a variety of 
perspectives. First, we will discuss the utility and impact of the Artful Learning professional 
development on teacher attitude and practice. Next, we will address the relationship of Artful 
Learning implementation to multiple classroom- and school-level outcomes. Also included in 
this section are findings regarding teacher-reported program impact. Finally, tentative 
findings regarding school level achievement data are presented. 

Professional development impact. Teachers’ overall impressions of the 
effectiveness/utility of the Artful Learning professional development were positive. A vast 
majority of the surveyed teachers felt that the professional development would help them 
both better implement the Artful Learning program (89%) and collaborate with other teachers 
(84%), as displayed in Figures 16 and 17, respectively. Teachers responded in a similar 
fashion to questions about the general quality of the sessions, with a vast majority reporting 
that the professional development offerings provided them with useful knowledge and skills, 
were well-linked to classroom practices, and were well-connected with each other. These 
responses are presented in Figure 18. 
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Figure 16. Professional development survey responses: Professional 
development will help me better implement the program. (N= 139) 
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Figure 17. Professional development survey responses: Professional 
development will help me better collaborate with other teachers. {N= 140) 




Provided me with Facilitators Sessions were 

knowledge or regularly made well-connected 

skills links 



Figure 18. Professional development quality: percentage of teachers 
responding “agree” or “strongly agree.” {N= 302) 

Teachers additionally reported how well the professional development prepared them to 
implement the various components of the Artful Learning program. Not surprisingly, there 
were differences in teachers responses based on the program component. Specifically, teachers 
were better prepared by the professional development for implementation of the classroom 
instruction aspects of the program than the organizational and, in particular, assessment aspects 
of the program. This finding was consistent for both Level 1 and Level 2 professional 
development participants, as presented in Figure 19. For example, 79% of Level 1 teachers 
reported that they were “adequately” or “more than adequately” prepared by the professional 
development for implementation of the classroom components of the program, but only 38% 
responded in a similar fashion regarding the assessment components of the program. 
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Figure 19. Professional development survey responses: How well did development prepare you to 
implement classroom component? (Level 1: N= 162; Level 2: N= 140). All percentages have been 
rounded up to the nearest whole number. 




Figure 20. Professional development survey responses: How well did development prepare you to 
implement organizational component? (Level 1: N= 161; Level 2: N= 138). All percentages have been 
rounded up to the nearest whole number. 




Figure 21. Percentage of teacher responses: How well did development prepare you to implement 
assessment component? (Level \ \ N= 151; Level 2: N= 140). All percentages have been rounded up to 
the nearest whole number. 



Analysis of qualitative data provided similar findings about the quality of the different 
components of the professional development offerings. In considering their Artful Learning 
professional development experiences, many Artful Learning-experienced teachers reported 
improvements in recent offerings compared to their earlier Artful Learning professional 
development experiences. Their initial training was cited as being too ambiguous and not 
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providing enough structure for teachers to implement the program with confidence and 
certainty. According to the interviewed teachers, the more recent professional development 
training addressed some of these concerns by providing clearer guidance. These reports 
suggest that recent changes made to the training by the Grammy Foundation were effective 
and improved upon earlier offerings. As one administrator stated: 

The professional development has been a little more concrete, and more specific to 
classroom activities. Yes, rather than being more theory-based, it’s been more about what 
can be done in the classroom, and also the focus is writing, so they’ve linked the program 
to more writing activities. 

The qualitative data also suggests that professional development had a significant 
positive relationship to program implementation, especially for new sites. New teachers were 
able to provide clear feedback on how the program is being implemented in their classroom. 
Many of the new teachers reported that they were excited about the program and exuded a 
level of confidence that was not evident with new participants from previous years. In 
particular, new teachers reported that they appreciated the opportunities to practice some of 
the strategies during their professional development training, which made it easier to return 
to the classroom and implement the strategy with students. 

I think it was also beneficial for us to actually go through the motions of becoming 
students and trying out the strategies ourselves rather than, “Let’s turn to page two, and 
look at this paper, and talk about it.” Or “Let’s look at this transparency. This is what the 
strategies are.” It was way more teacher involvement than a lot of other trainings I’ve 
been to. Force is a strong word, but when you do force teachers to try the strategies 
themselves, and become the students that they’re going to eventually teach those 
strategies — it allows you to own those strategies. 

I think it’s great the way (the professional development) is because it’s not only giving us 
new information and new strategies, and of course we practice it right there in class 
usually, but also the fact that we’re able to apply it to our classroom. It’s not just, “You 
can do this,” but we come back to the classroom and we actually implement it in our unit 
or in our regular everyday teaching. 

I think people have liked what they’ve seen. I think they see it as a really worthwhile way 
to look at teaching and learning for children. I think they’re feeling a lot of work. There 
is a lot of work to it because implementing something new takes a lot of time, but 
otherwise, overall, I think it’s really positive. I think for me it’s been really easy. 

Although their overall satisfaetion with the professional development was high, new 
and Artful Leaming-experieneed teaehers reported that there were still some eoncems with 
grade level and eontent area relevaney of the training. In particular, the lower grade level 
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teachers reported that many of the examples were geared for the primary school upper grade 
levels. Others reported that the training is geared for the language arts area, with not enough 
math and science examples. 

I think that for an introductory session, it needs to be brought down a little bit to what we 
can actually take and do right now. I know that they were showing us what it could look 
like in a few years, and especially a lot of the information was for the upper grades, so it 
was hard for me to walk away and say, “I can do this in my classroom.” It made me feel 
like I wasn’t sure that I could do it. 

They had shared Inquire Center for a lot of the upper grades of what an Inquire Center 
for the older children would look like, and it seemed really advanced. It was definitely 
not second grade, but I think I would’ve liked more of a primary example of an Inquire 
Center for next time. There are only so many minutes within a workshop that you can get 
everything, and. . . 

The presentation and the training were geared for the language arts and reading area, and 
so the other areas were wanting more examples of how to do that kind of thing with the 
math or science. 

Several teachers reported that because professional development was geared toward 
grade levels as well as content areas different from what they teach, they had difficulties in 
implementation. They found it challenging to transfer the information to their grade level or 
content area from the training when it came to implementing the program. 

The pre-post professional development surveys also questioned teachers about their 
perceived expertise about various instructional, assessment, and school-organizational 
activities. These items were used to create composite scales, as described in the Methodology 
section of this report, including a scale assessing instructional activities specific to the Artful 
Learning process, a scale focused on assessment, a scale focused on school 
organizational/decision-making activities, and a scale focused on general quality instructional 
practice (i.e., characteristics of quality instructional practice that are part of, but not unique 
to, the Artful Learning program). The SPSS General Linear Model function was used to run 
Multivariate Analyses of Variance (MANOVA) for both the Level 1 and Level 2 expertise 
scales.^ The overall multivariate Fs for both the Level 1 (F = 75.74) and Level 2 (F = 33.49) 
teachers were statistically significant,/) < .0001. The results for the individual scales show 
significant increases across all scales in teacher expertise after participation in the Artful 



* A MANOVA approach was used to conduct pre-test vs. post-test comparisons for multiple scales without 
inadvertently increasing the Type 1 error rate (i.e., the possibility that the statistical findings were due to a 
random error rather than representing true differences in the scores). 
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Learning professional development. In other words, teaehers’ reported knowledge and ability 
had increased across all aspects of the Artful Learning process (instructional, organizational, 
assessment) after participation in the professional development offerings. These results are 
presented in Tables 3 and 4 for the Level 1 and Level 2 teachers, respectively. 



Table 3 

MANOVA Results: Pre/post Level 1 Professional Development Survey Scales Change 



Scale 


Pre-test 

Mean/iD 


Post-test 

Mean/iD 


(Greenhouse- 
Geisser) F 


df 


P 


Artful Learning Process 


2.64/.90 


3.71/.59 


272.83 


1,111 


.0001 


Assessment 


3.04/.94 


3.65Z.82 


84.78 


1,111 


.0001 


Organization 


2.78/.88 


3.S8/.67 


133.51 


1,111 


.0001 


General Instructional 
Practices 


3.03/.85 


3.65/.71 


83.81 


1,111 


.0001 



Note. All scale item responses ranged from 1 {novice) to 5 {expert). 

Table 4 

MANOVA Results: Pre/post Level 2 Professional Development Survey Scales Change 



Scale 


Pre-test 

Mean/SD 


Post-test 

Mean/5D 


(Greenhouse- 
Geisser) F 


df 


P 


Artful Learning Process 


2.63/.79 


3.41/.57 


146.95 


1,86 


.0001 


Assessment 


2.60/.79 


3.39/.70 


101.65 


1,86 


.0001 


Organization 


2.64/.74 


3.43/.61 


104.31 


1,86 


.0001 


General Instructional Practices 


2.90/.78 


3.50/.70 


15.62 


1,86 


.0001 



Note. All scale item responses ranged from 1 {novice) to 5 {expert). 



Overall, teachers’ satisfaction with the Artful Learning professional development was 
fairly high in terms of both the content of the sessions and the processes for new teachers. 
For returning schools, teachers reported that as their concept understanding increased, they 
were able to get more out of the professional development training. In some cases, they were 
able to request professional development assistance based on the specific needs at their 
school site. 

Teacher impact. The vast majority of teachers reported that their attitude, and the 
attitude of other teachers at their school, toward the Artful Learning program is positive. 
Regardless of the time and effort that it takes to assign and complete a unit, many teachers 
reported that they are motivated to do whatever it takes to implement the Artful Learning 
program successfully. 
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The fifth grade completed their lesson plan. The way they presented it, the way the 
worked together with it, the fifth grade and kinder, they came together once they were 
totally excited about it. I think just from listening and observing, it even makes our 
curriculum, our state standards, it makes it easier to implement. When I say “easier,” I 
guess they’re just so excited about it, it seemed like there were so many positive attitudes 
about it. They were really, really excited about their lesson. They did theirs on the 
average man. 

As a result of the Artful Learning program professional development experience and 
subsequent unit development, new and Artful Learning-experienced teachers reported that 
their teaching practice is regularly influenced by the Artful Learning techniques. For 
example, one teacher reported that the tableaus are her vehicle for integrating the program 
across the different content areas. Many of the Artful Learning-experienced teachers reported 
that after their units are developed and implemented, they are inspired to reflect on how they 
can “perfecf ’ the unit for implementation the following year. 

I think that it definitely has forced me to rethink how to approach other units. We only 
decided to do the Level I implementation where you’re just planning for one unit. That 
was a lot of work to just plan for one unit. I think that it’s now making me think, “How 
can 1 make the other units better?” Like the next unit that we’re working on is about 
courage in Open Court, and so I’m beginning to think, “How can I make it as exciting as 
the fossil unit that we did?” It’s making me rethink how I’m going to approach that. 

Additionally, one administrator observed that the Artful Learning program calls upon 
teachers to “naturally” use different modalities for instruction, which in turn reaches learners 
with different learning styles. And finally, administrators reported that teachers are allowing 
students to take more control and ownership over their learning. 

They are able to teach all the different learning styles and reach more children. 

I think it’s allowed us to share ideas. Different ideas. We all have our own way of 
implementing lessons, but coming together just gave us the opportunity to share different 
ideas. Now I’ve used things that other teachers use in their classroom, and vice versa. It’s 
been a good experience. 

Implementation impact: Survey and interview findings. The teachers’ 

implementation survey provided several different types of information about the program 
impact on aspects of practice related to implementation. First of all, the survey provided 
information about factors related to greater quality Artful Learning implementation. As 
described in the Methodology section of this report, the implementation survey items were 
used to develop a series of scales focused on various school and classroom processes. In 
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order to approach the question of the relationship of teacher and school factors to quality of 
program implementation, we undertook a multiple regression analysis using scales from the 
implementation survey. 

Multiple regression analysis provides information about the independent relationships 
between an outcome factor and other factors that could be related to that outcome (referred to 
as predictors).^ In the case of this analysis, the outcome factor or variable was the Artful 
Learning Process Implementation scale. The predictors included factors/scales that would be 
hypothesized to be related to higher levels of Artful Learning implementation; that is, schools 
fully using the Artful Learning model would be expected to excel in these other areas as well. 
These predictor scales included quality instructional practices, assessment use/understanding, 
standards use/understanding, shared leadership, shared school mission, parent involvement, 
and perceived program impact. In addition to these scales, predictors for this analysis also 
included, as controls, the amount of time the school has been involved in the Artful Learning 
program and years of prior teaching experience. 

The statistic from the analysis, .48, indicated that approximately 48% of the variance 
in the Artful Learning Process Implementation scale could be explained by the other 
variables, with the relationship of the Artful Learning Process Implementation scale to the 
predictor variables overall being statically significant: F(9,255) = 25.95, p < .0001. Beyond 
this overall relationship, the relationship between the Artful Learning Process 
Implementation scale and the individual variable were analyzed, as presented in Table 5. 
These results indicate that some, but not all, of the predictor variables had a statistically 
significant independent relationship with implementation quality. 



^In the case of multiple predictor factors, multiple regression analyses the relationship between each predictor factor and the 
outcome factor while controlling for possible relationships between the other predictor factors. In addition to providing this 
statistical control to allow for simultaneous multiple comparisons, the multiple regression approach avoids increased error 
problems that could emerge from running multiple individual analyses. Additional information about multiple regression 
can be found in Tabachnick and Fidell, (2000). 
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Table 5 

Results of Multiple Regression Analysis: Artful Learning Process Implementation as Outcome Variable 



Scale 


B 


Std. Error 


Standardized 

Beta 


t 


R 


Years Teaching Experience 


.034 


.021 


.075 


1.59 


.114 


Years Using Artful Learning 


.070 


.038 


.090 


1.83 


.068 


Assessment Use/Understanding 


.118 


.056 


.117 


2.13 


.035 


Standards Use/Understanding 


.104 


.049 


.102 


2.14 


.033 


Reported Impact 


.208 


.043 


.245 


4.87 


.000 


Shared Leadership 


.102 


.062 


.107 


1.63 


.104 


Quality Instructional Practices 


.134 


.054 


.131 


2.47 


.014 


Parent Involvement 


.090 


.047 


.105 


1.91 


.057 


Shared Mission 


.197 


.050 


.235 


3.94 


.000 



As the table shows, five faetors had a significant independent relationship with the 
quality of implementation. Higher levels of implementation were related to great use of 
quality instructional practices in general (i.e., not specific to the Artful Learning program), 
greater assessment use/understanding and greater use/understanding of state standards. 
Teachers who implemented the Artful Learning classroom components also reported a 
greater sense of shared mission/goals at their school than teachers with lower levels of 
implementation. Not surprisingly, those teachers who were implementing the program at 
higher levels also reported greater program impact than those with lower levels of 
implementation. In other words, program implementation was related to a number of positive 
classroom, school-level, and impact variables, with greater implementation indicating higher 
levels of these factors. The relationship between Artful Learning and assessment in particular 
suggests that while, as described elsewhere in this report, teachers are still experiencing 
difficulties with the assessment components of the program, they may be beginning to make 
at least some assessment links to their Artful Learning instruction. 

It is also important to note that several variables did not have a significant independent 
relationship with Artful Learning implementation. There was significant relationship between 
Artful Learning and parent involvement, consistent with the limited parent involvement 
connections to Artful Learning described in the interviews. However, of the few interviewed 
teachers who reported on the impact that the Artful Learning program has had on parents, it 
is important to note that the impact is positive. Teachers spoke of the value and of the effort 



29 









made by their sehools to inform and invite parents to partieipate in the Artful Learning 
process so that it can he supported at home. 

You would not believe how many parents get involved in these projects that we do at the 
school. They come in and they’re just raving about it. That’s one of the things that I see, 
and that’s why I think our auction is going so well is because I think the parents value 
this. They say, “I want my child excited at school. They’re happy. They’re coming home 
talking about it. That’s what I want to see.” I think that’s what’s happening. 

Our parenting programs that we offer are also presented using the Artful Learning 
program. So it helps parents understand better the process that their students are going 
through. I think the parents have also seen an increase in the enjoyment of learning by 
their students and in the level of learning. 

There also was no significant relationship found on the survey between the level of 
shared leadership at the school and Artful Learning implementation. This finding was 
unexpected, given the emphasis on school-level collaboration in the Artful Learning 
program. However, based on qualitative data it seems that many of the organizational 
processes at the school — such as decision making and leadership — are constrained by school 
and district factors outside of the Artful Learning program, and thus may be resistant to the 
influence of the Artful Learning implementation process. 

It is also worth noting that neither teacher experience nor years of involvement in the 
Artful Learning program were significant predictors of Artful Learning classroom 
implementation. This finding suggests that, even though the program can be challenging to 
implement, teachers with a variety of years of teaching experience are capable of using the 
program consistently. The lack of relationship to years involvement in the Artful Learning 
program is not surprising as based on qualitative data, there is no apparent relationship 
between a schools’ amount of participation in the program and approach to scale up. That is, 
there are some older schools that have been slower/less complete in their implementation and 
use of the program, and newer schools that have been quick to implement the program 
intensively throughout the school. This suggests that, even though the program can be 
challenging to implement, teachers with a variety of years of teaching experience are capable 
of using the program consistently. 

The quantitative and qualitative data also provided information about specific areas of 
program impact on students. In the implementation survey, over 70% of teachers reported 
that the Artful Learning program had moderately or greatly increased their students’ level of 
engagement/interest with the curriculum. These results are shown in Figure 22 
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Figure 22. Teacher survey responses: “moderately” and “greatly increased”: 
Impact of Artful Learning. 



Similarly, interviewed teaehers and administrators highlighted student engagement and 
interest as a key impaet of the program. As teacher practice through the Artful Learning 
program called upon the use of different modalities for instruction (e.g., visual, auditory, 
fine-motor skill activities, and so forth), it resulted in more rigorous instruction and increased 
student engagement. That is, students received information multiple times and in multiple 
ways, and this seemed to give many students the ability to “access” or retain and process 
information using higher level thinking skills. 

I think it’s had a tremendous impact in that I think that the students are more engaged in 
their learning. I’m able to use more rigorous instruction with them because they are 
experiencing more of the information instead of just having it spewed out at them and 
having them tell it back. They are processing and applying it using higher level thinking 
skills, I think too, because they are able to “see” the connection. 

They do listen and they are interested and they do remember and they have told me 
things from last year that I would have thought that they did not get. I thought, “Oh, I did 
a lousy job and they didn’t understand that.” Then, this year I hear, “I remember 
when...” and I thought, ‘Whoa! You were listening.’ 

Their enthusiasm skyrocketed. Their participation. The quality of work definitely. I think 
in this you give a certain amount of control to the students. It’s not that classic 
educational model or direct instruction. 



In terms of academic impacts, approximately 60% of the surveyed teachers reported 
moderate or great increases in the quality of students work due to their participation in the 
Artful Learning program. A slightly smaller percentage of teachers responded similarly 
regarding students scores on classroom assessments and standardized tests, 48% and 47%, 
respectively. These findings are displayed in Figure 22. However, it is important to note that 
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30% of teachers reported that they did not know if the program had any impact on student 
standardized tests. This finding is again consistent with some of the ongoing concerns 
teachers have reported about linking assessments to their Artful Learning practices 
highlighted elsewhere in this report. 

The qualitative data supports the findings from the teacher survey regarding 
considering the quality of student work. Teachers reported that students were exposed to 
many more opportunities for higher-level thought and analysis than they would have been 
exposed to under normal school circumstances, and through these exposures greater learning 
is taking place. For example, as one teacher reflected on her experience with students from 
other school sites, she found that “Artful Learning” students are better able to articulate 
information and are more accepting of cultural differences than students from other school 
sites. As teachers described: 

I’ve seen writing in the past in this district from as far back as 10 or 15 years. I’m talking 
about a broader spectrum. Instead of saying just simplistic things like, “I like that,” they 
are more involved with their writing. I think they are more so than I’ve seen at any 
school. I told that to the principal when I came here. I said, “The writing is just 
unbelievable.” I think that you can’t say it’s any one particular thing, but I would say 
very strongly that Bernstein (Artful Learning) has a major, major impact across the 
curriculum. 

They are more articulate. They are more accepting of different cultural introductions. For 
instance, if I were to put a picture of an African mask up, they wouldn’t necessarily burst 
into laughter. They would find it intriguing, and they would appreciate the beauty, and 
they would already have some prior knowledge perhaps of why the markings are as they 
are. They can have a higher level of discussion about a piece of art. They’re patient 
enough and sophisticated enough to engage in Inquire. Whereas some of the other 
students because they have not been exposed culturally they’re first reaction quite 
possibly would be to laugh because they haven’t had the exposure. 

Although both teachers and administrators are seeing student outcomes improve, they 
admit that they caimot attribute these gains solely to the Artful Learning program due to the 
absence of an Artful Learning-focused classroom assessment. They are confident in their 
speculation, however, that the Artful Learning program has played a major role in increasing 
student scores. 

I don’t know that we’ve come up with a tool to assess that, but we certainly have a record 
of our test scores, and our site does well. We have definitely shown that. We’ve had a 
couple of dips, but we’ve had more increases than dips since the model has started... 
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School-level achievement data. As described in the Methodology section of this 
report, the only standardized student achievement data available for our evaluation was 
aggregate, school-level data. There are several limitations to using this type of data. The lack 
of linked student- and teacher-level data makes it very difficult to identify achievement 
change and attribute it to Artful Learning program participation, particularly because many 
of the schools have some teachers who have yet to implement the program in their 
classrooms. The lack of individual data also makes it impossible to track individual students 
over time; rather it only allows for comparisons of successive cohort groups of students; for 
example, comparing the achievement of Grade 3 students in the year 2002 with Grade 3 
students in the year 2003. Such a comparison does not take into account possible changes in 
the grade’s student population, demographics, or prior academic achievement. Furthermore, 
in terms of Artful Learning, this type of data does not incorporate any information about how 
long the students included in each cohort have been participating in the Artful Learning 
program. It should be noted also that the Artful Learning schools are located in a variety of 
states, each with their own state standards and standards-based achievement tests. 
Additionally, the nature and methodological limitations of the school-level data do not allow 
for running of inferential statistics to determine statistical significance of any differences 
found. 

With these caveats in mind, we present descriptive information about changes in 
student achievement at the Artful Learning schools. Specifically, we compared the growth in 
students meeting or exceeding state standards at the Artful Learning schools to their districts 
as a whole and a matched sample of comparison schools. We focused on schools that had 
been involved in Artful Learning at least 3 years, were currently still using the program, and 
had 3 years worth of school achievement data publicly available up to 2003. A total of nine 
Artful Learning schools met these criteria. As described in the Methodology section, the 
decision was made to focus on Grade 4 for the primary schools and Grade 8 for the middle 
schools, as these were the commonly tracked grades prior to the advent of No Child Left 
Behind (NCLB) legislation reporting requirements in 2001.^ 

Table 6 displays the average growth over 3 years in the percentage of students 
meeting/exceeding state standards'^ for the nine Artful Learning schools, their districts as a 
whole, and 18 matched comparison schools (two matched comparison schools for each 
Artful Learning school). Results are presented for reading, language arts, and math; note that 

^ NCLB is federal legislation requiring all schools to test and report test results for all Grades 3-8, among 
numerous other reporting requirements. 

* The states differed in the terms they used for these designations, but all states had categories for students who 
had met or exceeding grade level requirements. 
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due to state testing policies, five of the Artful Learning schools provided separate scores for 
both reading and English language arts, two schools provided results for reading only, and 
two schools provided results for English language arts only. All of the schools provided math 
scores. 



Table 6 

Average Growth in Percentage of Students Meeting/Exceeding State Standards for Artful 
Learning Schools, their Districts, and Matched Comparison Schools (2000-2003) 





Reading 


ELA 


Math 


Artful Learning Schools 


11.5 


5.4 


9.7 


District 


5.5 


5.2 


5.9 


Comparison Schools 


4.9 


3 


5.6 



Note. Data sources: State Department of Education websites for CA, FL, GA, PA, and OR. 



As the Table 7 shows, on average the Artful Eeaming schools showed a larger growth 
in the percentage of students meeting and exceeding standards than either their districts as a 
whole or their matched comparison schools for reading and math, and showed a larger 
growth than their matched comparison schools in English language arts. When broken down 
year-by-year, this trend appears most consistent for reading and math; with the Artful 
Eeaming schools on average out-performing in growth their districts as a whole and their 
matched comparison schools for each of the years included in the analysis, as shown in Table 
6. Eor example, even when, on average, there were dips in the percentage of students meeting 
and exceeding standards year-to-year, these dips were generally less for the Artful Eeaming 
schools than for their comparison schools or districts as a whole. 
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Table 7 



Average Growth in Percentage of Students Meeting/Exceeding State Standards by School 
Year for Artful Learning Schools, their Districts, and Matched Comparison Schools 



Year 


Artful Learning 


District 


Comparison 


Reading 


2000-’01 


17.1 


10.5 


14 


2001-’02 


1.5 


1.5 


-2.5 


2002-’03 


-1.4 


-3.8 


-3.5 


ELA 


2000-’01 


8.6 


5.2 


5.1 


2001-’02 


1 


2.5 


-2.4 


2002-’03 


-1.8 


-1 


-1 


Math 


2000-’01 


8.1 


3.8 


4.7 


2001-’02 


2.1 


2.0 


2.5 


2002-’03 


3.6 


2 


2 



Note. Data sources: State Department of Education websites for CA, FL, GA, PA, and OR. 



As noted elsewhere in this report, these results should be interpreted with caution due 
to the methodical considerations outlined. These findings alone are not sufficient as evidence 
of a direct, specific impact of the Artful Learning program on student achievement, which 
would require the analysis of individual student level data with matched control student 
samples. However, these descriptive results do provide some tentative positive information 
regarding student achievement in the Artful Learning schools, and emphasize the importance 
of exploring possible avenues for analyzing, with district approval, student level data. 

A smaller number (four) of Artful Learning schools also have also collected several years 
writing assessments as part of their state and/or district’s assessment plan. Due to changes in 
state and district reporting, data was only available for all of these schools for 2000, 2001, and 
2002. Figure 23 shows changes over time in the percentage of students at the two highest 
writing assessment designations for these four Artful Learning schools and their districts as a 
whole. 
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Figure 23. Change in percentage of students at the two highest writing assessment categories for four Artful 
Learning schools and their districts. 



Again, although the data is anecdotal, there are again examples of greater growth in the 
Artful Learning schools than the districts as a whole. Each of the schools showed 
comparatively greater growth from 2000 to 2002 than its larger district; on average, the 
Artful Learning schools grew 22% in the students at the two highest designations of writing 
assessment, while the district only grew an average of 12%. Writing is a particularly 
important skill area within the Artful Learning framework, as writing-based activities are 
often used as a bridge between arts and the other content areas, especially as part of the 
Reflect component of the Artful Learning model. Writing assessment data from additional 
Artful Learning schools would thus be a very useful piece information in terms of the overall 
achievement impact picture for the program. 

Facilitators and Barriers to Implementation 

Across the years of CRESST’s Artful Learning evaluation, several themes have 
emerged from the qualitative data regarding factors that have facilitated program 
implementation and barriers or roadblocks that have interfered with the implementation 
process. This year, as in years previous, implementers identified several variables that had 
the potential to either help or hinder the schools’ overall implementation process. The 
“veteran” Artful Learning teachers, those who have been using the program for several years, 
had some particularly interesting insights about this process. Lor the purposes of this 
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discussion, the variables are divided into three general categories: resources, materials, and 
time; structures for support; and district/state requirements. 

Resources, materials, and time. Many teachers reported that aecess to materials, 
including Masterworks, was a barrier to implementation. This diffieulty included financial 
concerns for purchasing materials as well as problems loeating or gaining aceess to multiple 
eopies of a Masterwork. As diseussed earlier in this report, many of the teachers are highly 
motivated and exeited about the Artful Learning program, but the struggle to gain access to 
materials can compromise implementation as teaehers grow tired of trying to access the 
neeessary materials for implementation. 

Our team, unfortunately, has been having a hard time finding multiple copies of our 
Masterworks. I’m not sure how we’re going to work around that. ...(the Grammy 
Foundation) has encouraged us not to give up, and that we’ll find the resources. We’ve 
only been able to find our Masterwork sold as part of a set of multiple posters. 

I think there were times where I felt like materials, and even though the school said 
they’d pay for everything, that’s still hard to have happen where it’s like you have to tell 
them way in advance what you need to buy. It’s so much easier to just go out by yourself 
and get the materials. That can be a process of running around. I know my friend, and the 
other second grade teachers had a lot of materials to gather if they were teaching the art 
portion or the science portion. So probably the roadblock would be gathering the 
materials. 

Both new and Artful Learning-experienced teachers reported that finding the time to 
develop the units during the school day continues to be a concern. In particular, at school 
sites where implementing the Artful Learning program is voluntary, teachers who do not 
initially implement the program see the amount of time that their colleagues who do 
implement devote to the program as a deterrent. That is, if implementation is voluntary, the 
time devoted by the first “wave” of implementation ean serve a disineentive for later 
implementation and scale up to additional teachers. 

The only thing that I’ve really heard as far as a concern is about the intense amount of 
planning time that it’s taking us to build our unit. 

I guess whaf s been most difficult is just the coming together. Just for us to have that 
time to sit down, and just talk — communicate. 

Intense and long. Last year we planned together.. .on our own time. We do not have 
common planning, so we would always talk in snatches on our lunch break, snatches, in the 
morning, and then when we really got serious this year, we would stay after school 
everyday at least once or twice a week for hours — for hours. Other teachers recognized, 
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“You must have to put a lot more work into Artful Learning. I don’t want to put in more 
hours, so I don’t want to do it.” 

Many of the new teachers expressed concerns about finding sufficient class time to 
implement their Artful Learning unit. These teachers also reported that it was difficult for 
them to determine the amount of class time that should be dedicated to each of the four 
components. 

It’s been difficult because of time. We’re so short on time that they are not being given 
the opportunity to have enough time to explore. That’s what I feel. It tends to be, the 
whole group lesson I introduced it as an Inquire, and then when we go to the center they 
can’t spend a lot of time there. 

Most of the difficulty was in, number one, the amount of time it takes. Time management 
was a big thing. The first year, I did something similar to it, but my time management 
was so bad that I had to cut back on the number of pieces the students had to produce. I 
don’t think they really got the full impact of the connection to the literature because all of 
the literature was set in the 1940s. Time management was difficult. 

Structures for support. There were several additional sources of implementation 
support that either facilitated or, by their absence interfered with the implementation process. 
Overall, teachers reported that, to varying degrees, their school administration were 
supportive of the Artful Learning program. For example, just over three-quarters of the 
surveyed teachers “agreed” or “strongly agreed” that their administrators supported Artful 
Learning implementation, as shown in Figure 24. Interviewed teachers similarity highlighted 
the presence and importance of support from their administrators in the overall 
implementation process. 

She is always in there positive, positive, positive. She never puts anybody down. I think 
for this research, and whatever the conclusion of this is, that this program would never be 
as successful as it is without the leadership. If you didn’t have a principal so totally into 
the arts, so open to taking a risk, I mean she’s taken big risks and gotten slapped on her 
fingers. “You can’t do that.” She takes the risks, but look at the payoff. If s just great. 

She’s excited about it. As far as being able to go up and ask her questions about it, I don’t 
think that she has enough knowledge of the program yet. Now the vice principal she has 
a lot more knowledge than the actual principal. She’s ready to share. She’s the one that 
opened the school up on Saturdays for us to have the time to come in and do this. 



38 




ioo%n 



80% 




Strongly Disagree Disagree Agree Agree Strongly 

disagree slightly slightly agree 

more than more than 
agree disagree 



Figure 24. Teacher responses: There is support for Artful Learning 
implementation from administrators at my school. (N= 272) 



Beyond administrator support, the availability of an Artful Learning eoach appeared to 
play an important role in supporting the implementation process. Those teachers who had a 
designated Artful Learning coach at their school highly valued the support that was provided 
by these coaches for implementation. Among surveyed teachers who had a coach, the vast 
majority rated the myriad support offered as “useful” or “very useful,” as shown in Figure 
25. The Artful Learning coaching activities of team building, assistance in unit development, 
and supplementary group professional development seminars were rated particularly highly. 
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Figure 25. Teachers responding “useful” or “very useful”: Types of coaching 
support. (A= 251) 

The interview data similarly indicated that those teachers who had access to coaches 
saw them as an important supportive factor in the implementation process. As one teacher 
with previous Artful Learning implementation experience described the importance of her 
coach: 
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One thing, if there’s anything you need other than money she’s [Artful Learning coach] 
there. She’ll go out and help you find Masterworks. She’ll help you find whatever you 
need to support the lesson. If you need sample lessons she tries to help with that. She had 
a little survey that she passed out earlier. It says, “What are you needs? I’m available to 
do this.” Quite a few people took her up on it. She is there in a supportive way and in a 
leadership way. 1 told you the things we did at the end of last year. She was in charge of 
getting all of that done. She pulled the teachers together in a supportive way and in a 
leadership way. Maybe you’re not sure you’re doing something correctly, and maybe you 
need to talk it over with her to see if you’re on the right track she’s there. Also, she let the 
community know what we were doing, and she had the community involved. 

Although those teachers who worked with an Artful Learning coach described it as an 
invaluable part of the implementation process, many teachers reported that they do not have 
an Artful Learning coach and/or were confused about what an Artful Learning coach was and 
what they did. Many of the interviewed teachers were either unaware that their school 
actually had a coach, or assumed that a designated grade level leader as the “coach.” 
Similarly, when asked if they had a Artful Learning coach at their school, almost 20% of the 
surveyed teachers said “no,” and an additional 14% did “not know” if they did or not. These 
results are shown in Figure 26. 




Figure 26. Teacher survey responses: Presence of an Artful 
Learning coach. {N= 367) 



As described in earlier sections of this report, there was very little impact of the Artful 
Learning implementation on parent and community involvement. However, there were some 
anecdotal examples of ways that parent resources have been tapped on to facilitate program 
implementation and perseverance. For example, one of the more experienced sites reported 
how they involved the community to raise funds to financially sustain the Artful Learning 
program at their school site, as district money becomes tougher to get. This particular school 
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site holds an annual auction. Each year, the fundraising goal has increased with hopes of 
attaining the same amount that was used to initially implement the program. Such an 
approach not only helped raise funds for the school, but also promoted parent “buy-in” to the 
program. 

Yes, and that was real key to have the time and the funding to be able to do that. For a 
while, in the middle of our 6- or 7-year period, we didn’t have the funding, and we were 
kind of doing it on our own. Finally, we started this auction, and that has brought more 
funding in, so the last couple of years people are getting more excited about it because of 
this new funding that we have now. We’re going for more this year. 

District and state requirements. A final key variable that has either facilitated or 
impeded Artful Learning implementation are state/district mandates and requirements. 
Several of the new and Artful Learning-experienced teachers reported that they had concerns 
about competing with curricula mandates and trying to incorporate the Artful Learning 
program into these mandates. As more and more states and districts align standards, 
assessments, and curricula, and publishing companies follow suit by providing new 
standards-based curricula textbooks and materials, principals and school districts are 
encouraged or sometimes mandated to adopt these new standards-based textbooks and 
materials.^ The Artful Learning participants are impacted because teachers are often being 
asked to shift their attention to implementing these new comprehensive curricula, that, on 
face value at least, are not always consonant with the Artful Learning model. Likewise, many 
administrators described the rigorous mandates set forth by the district that must be adhered 
to, and how these mandates interfered with their school site’s ability to consistently 
implement the Artful Learning program. 

Our district has a very structured and rigorous curriculum that teachers have to adhere to, 
so it’s difficult to engage in Bernstein (Artful Learning) activities. Teachers really have 
to make it a priority to include the program. It’s a challenge. 

It was much harder this year because we had a new staff, and we had three or four new 
programs that they said we must do. 

The one thing is that (Artful Learning) requires a little more time than what we have, and 
maybe this is our design flaw in creating our Inquire Centers than what you have time 
for. They have time for it, so that adjustment hasn’t been bad. Maybe when we designed 
it, too, I think we took that into consideration of, “Let’s just fit this into our normal day,” 
because we couldn’t upset our normal day. 



^ Currently, 2 1 states have a formal textbook adoption policy in place, requiring districts to select their curriculum from a list 
of approved published programs. 
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Interestingly, some of the Artful Leaming-experieneed teachers reported that although 
they are mandated to use standards-based curricula, as they have begun using the Artful 
Learning program they have found it enriches their teaching and brings more life to their 
mandated curricula. They are thus optimistic about finding ways to implement both the 
mandated curricula and the Artful Learning program. These teachers reported that although 
there were initial struggles, they were able to eventually integrate both the mandated 
curricula with Artful Learning, and this was by choice. That is, teachers reported that there 
was value added in using both programs. 

I tried a couple of the different strategies. I tried the tableau. They enjoyed that. They did 
a reader speeder for one of the stories within the fossils unit. They got a lot out of that. I 
tried to incorporate dinosaur rap songs and they really liked that. For hands on activities, 
they got to become scientists, and open up owl pallets, and take out bones from the owl 
pallet, and recreate a skeleton. They had literature, and more literature to read about 
dinosaurs. Between all of our mixing activities, they were making soil in one of the 
groups with real getting your hands all dirty. The social studies that I did with my 
students was good because it was also incorporating personal objectives including some 
of the Artful Learning strategies. I guess I’m being kind of vague, but I feel like they 
were so much more excited about this unit. 1 don’t think it’s just because it was 
dinosaurs, I think it’s because we really tried to just expand beyond the dry Open Court 
reading. 

Not surprisingly, given increasing federal testing requirements associated with NCLB 
legislation, many Artful Learning participants feel that state and district assessment 
requirements also interfere with consistent and persistent implementation of the Artful 
Learning program. The amount of time spent on testing activities and test preparation 
distracted from these teachers’ Artful Learning units, and their attempts to view their 
instruction through the lenses of Artful Learning. As two teachers described: 

I happen to facilitate the fourth grade/grade level team, and as far as any barriers that I 
see, it happens regardless of what you’re doing, and it doesn’t have anything to do with 
this arts program. It’s the overburden and pressure that’s put on teachers across the nation 
because of standardized testing. 

As far as the Rubrics we use, they are the Rubrics our county requires us to use. So for 
the language arts teachers there are Rubrics that are required from my county for me to 
use for sixth grade writing samples, and seventh and eighth grades, so those are the 
Rubrics that I use for all of my writing assignments. 
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On the other hand, a smaller number of distriets, rather than interfering with the Artful 
Learning implementation proeess, were taking formal steps to support the implementation of 
the program. For example: 

Our Superintendent is trying to provide those collaboration times for us to work together. 

My team, the second grade team, we get together at those collaboration times, and we go 
over our plan for our units. We’ve been able to use that as sort of the preparation time, 
but I know that people are a little nervous as far as the intensity that’s involved in this. 

It’s pretty extensive, and I think people are concerned about, “How much of the work 
that I’m doing in the classroom, and how much of the time do I just need in prep am I 
having to give up or having to re-focus towards the developing the Bernstein (Artful 
Learning) model?” Outside of that, I would say most people are very excited about it. 

For the majority of teachers and school administrators, however, the districts were at 
best a non-entity in the implementation process: not providing any additional support but not 
creating any insurmountable roadblock. The question of fitting Artful Learning in among 
growing state and district mandates remains a concern among the schools in terms of the 
long-term sustainability of the program. 

Conclusions/Recommendations 

Following is a summary of the evaluation results presented in this report. The results 
are placed in the context of the whole sum of evaluations findings from CRESST previous 
data collection years. This section concludes with a series of suggestions/recommendations 
for the continued improvement and refinement of the Artful Learning program. 

Summary/Conclusions 

The findings presented in this report represent the final year’s data from CRESST’s 
3 -year role as external evaluator for the Artful Eeaming program. It is therefore useful to 
view these findings within the context of previous years’ findings. Many aspects of the Artful 
Learning participants’ experiences this year are consistent with the findings in earlier years’ 
evaluation reports (Griffin & Schwartz, 2002; Griffin, Miyoshi, & Kao, 2003). Over all years 
of the evaluation, participants have been extremely pleased with the professional 
development that they receive from the Grammy Foundation, and these positive evaluations 
have continued to grow as the Foundation has revised and refined the professional 
development offerings over the years. Furthermore, as the schools have continued their 
participation in the program and received additional levels of training, this satisfaction has 
continued, with teachers consistently pleased with all levels of the professional development. 
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Across the years, there has also been consistent satisfaction with and persistence of use 
of the program on the part of the teachers, even in the face of some of the potential 
roadbloeks deseribed in this report and previous years’ reports. Across the schools, teachers 
have eontinued to design units that integrate the Artful Learning process aeross multiple 
eontent areas. Aeross all years of data, similar eomponents of the program have emerged as 
eausing the most diffieulty to teaehers in implementation, ineluding Reflect and Inquire 
aetivities and the integration of the model into subjeets matter not traditionally linked to art 
(i.e., scienee and math). At the same time, the quality and ereativity of the Artful Learning 
units that teachers are designing seems to have grown over time. Teachers are tapping on a 
more diverse spectrum of masterworks as they eontinue to implement the program, and are 
more apt to deseribe a well-thought out process of seleeting a masterwork that fits their 
eurriculum content than in earlier implementation years, as opposed to relying on the limited 
set of masterwork examples provided in the Grammy professional development offerings. 

Aeross the evaluation years, assessment has also continued to be a difficultly for 
participating teachers. The assessment needs reported by teaehers have grown to inelude both 
issues relating to student assessment and those relating to self-assessment of instruetional 
quality. Basically, teachers remain unsure about exactly what they should be doing to assess 
students within the Artful Learning framework, and how to use whatever information they 
gather from the assessments. Given the growing state and district assessment demands that 
many of the sehools are faced with, a more pertinent question might be how to integrate 
Artful Learning-focused assessment into the sehools’ existing, mandated assessment systems. 
This year, the teaehers appeared to be taking greater steps towards sueh integration, as 
evidenced by both survey and interview findings. However, assessment remains a diffieult 
eomponent of the model overall to teachers, who appreeiate the eritical role of assessment in 
terms of both guiding their instructional practice and demonstrating student suceess. 

It is important to note that assessment has also been tied to issues of program impaet 
throughout the years of this evaluation. Throughout the years, reported program impact from 
participants has been high, partieularly in terms of student engagement and classroom 
learning. Although their assessments of program impaet on student achievement has been 
relatively high as well, a signifieant number of teachers remain unsure about impaet in this 
area, predominantly due to the faet that they are not actually sure how to assess it. For this 
year’s evaluation report, adequate data was available to consider school-level ehanges in 
aehievement for the Artful Learning sehools in comparison to schools not using the program, 
with some positive trends emerging. Again, although these trends provide some important 
initial information, they also leave many information gaps that ean only be filled by analysis 
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of student-level data. So, both at the individual teacher level and the program-wide level, 
availability and use of various types of assessment information remains an issue requiring 
on-going attention. 

Finally, throughout the years of the evaluation some consistent trends have emerged in 
terms of the potential barriers to program implementation that participating schools have 
faced. Specifically, the role of the larger school district and district mandates has been 
prominent throughout the years of the evaluation, as has continued to grow with the 
increased demands placed on states due to No Child Left Behind (NCLB, 2002). Regardless 
of the amount of district buy-in to the Artful Learning program, each school has had to 
address the initial task of fitting the Artful Learning instruction and its time demands, both in 
terms of classroom time and time for planning, into the overall demands of district-wide 
curriculum and assessment requirements. For schools with mandatory textbook adoptions® 
this task has been particularly challenging, attempting to integrate Artful Learning instruction 
into pre-existing curricular packages that have been both designed to stand on their own and 
structured to consume all of the school day time set aside for a particular subject area. 
Looking back over the 3 years data, most of the teachers facing textbook adoption, although 
initially struggling, have found ways to incorporate Artful Learning into the mandated 
programs that they are using. However, the process of integration and role of the Artful 
Learning curriculum at these schools is somewhat different from that at schools that do not 
have a program adoption but have the autonomy to create their own curriculum. 

In summary, throughout this evaluation the Artful Learning program has shown to be a 
useful tool for teachers with a variety of previous teaching experience, district and state 
contextual demands, grade/content areas taught, and student populations. The professional 
development support provided to schools through the Grammy Foundation has been effective 
for the teachers, and the foundation has continued to adapt and fine-tune the training based 
on feedback and the needs of the schools. However, there is still potential for continued 
growth and improvement, both in terms of schools using the program and in the program 
organization/design itself With that in mind, based on this year’s evaluation data in 
integration with data from the evaluation as a whole, we provide some recommendations for 
the future of the Artful Learning program. 



Currently, 20 states have mandatory textbook adoption policies; that is, districts in the state must use one of a 
list of approved published curricula for a specific content area. The majority of these adoptions are in language 
arts and math, although states are beginning to implement similar policies for science as well. 
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Recommendations 



Recommendations can be divided into three broad categories: training and support, 
assessment, and evaluation considerations. 

Training and support. As described elsewhere in this report. Artful Learning teachers 
were very satisfied with the professional development provided by the Grammy Foundation, 
as well as with the level of support provided by their school administrators. However, there 
are some additional areas of support that could be developed to assist teachers in the on- 
going implementation process. These types of support include: 

• Better use/integration of unit examples: Teachers and schools should provide more 
specific guidelines for developing unit components, stressing the importance of 
implementing all four components {Experience, Inquire, Create, and Reflect). For 
example, there should be time available during all levels of professional 
development for teachers to develop all four units, with multiple examples provided 
from a variety of artistic modalities. Many teachers need to see unit examples to use 
as a guide during their first years of developing units. Although this may 
compromise creativity, it may actually strengthen their units over time. 

• Eye towards curricular adoption: As noted previously, the experiences of teachers 
who are trying to implement Artful Learning in the context of core state-wide 
curriculum are different from those teachers who are not subject to a required state 
or district-wide curriculum. Furthermore, the processes and products of designing 
units look very different for the teachers using a state-adopted published curriculum 
compared to those who do not. Such issues could be integrated into the Artful 
Learning professional development and materials, with examples provided from 
schools and classrooms that have successfully implemented Artful Learning units 
within an adopted materials context. 

• Better orientation/support for website use: The Artful Learning website is an 
incredible resource for dissemination and communication between Artful Learning 
schools and participants. However, teachers are not making the use of it that they 
could, due to a combination of lack of information and lack of technical comfort. 
More experience/practice with the Artful Learning website could be built into the 
training activities, both to orient teachers to the website and to give them the hands- 
on opportunity to access the types of resources that are available there. 

• Renewed focus on art and music teachers: The training should continue to 
emphasize the value in using art and music teachers to support teachers with 
suggestions, integration across content areas, and so forth. These teachers should be 
included in all phases of the Artful Learning professional development for all of the 
participating schools. 

• Consistency of coaches: Those teachers who worked with Artful Learning coaches 
found the experience very helpful in terms of the implementation process. However, 
many teachers either did not have a coach at their school or were not aware that 
there was a coach. Attempts should be made to bolster the coaching process, both in 
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terms of emphasizing the importance to schools of having an Artful Learning coach, 
and working with the schools to best integrate the coach into the day-to-day 
workings of the school. 

Assessment. The integration of assessment into the Artful Learning program emerged 
as an area of some difficulty throughout the evaluation process. Although some of the more 
experienced teachers are beginning to find ways to address assessment within the Artful 
Learning classroom, many teachers still struggle with this issue, both in terms of assessing 
their students and their own work. Some suggestions for ways to better support teachers in 
the assessment process include: 

• Develop common assessments: One on-going concern on the part of teachers has 
been how to approach classroom assessment as part of their Artful Learning units, in 
order to determine student progress and guide their instruction. Although the Artful 
Learning rubrics themselves are a useful base to build on, they do not seem 
sufficient for most teachers, particularly those in the earlier phases of Artful 
Learning implementation. Teachers might be assisted by a common assessment 
strategy and procedure that they could then adapt to each of their Artful Learning 
units to develop individual assessments tied to each unit. These individual 
assessments could then be shared along with the units on the Artful Learning 
website. Furthermore, should resources become available, the Artful Learning 
program itself could develop a common set of performance-based assessments at 
key grade levels for teachers to use in order to monitor student progress within the 
Artful Learning framework. 

• Develop structured materials for self-assessment: There was a great deal of 
uncertainty among teachers regarding the quality of the Artful Learning classroom 
processes, and how to determine if they were effectively implementing the program. 
Teachers reported a desire for additional tools to assist them in this process, such as 
a self-assessment rubric or checklist, as well as some guidance in the process of 
self-assessment. These materials and supports, once designed, could be easily 
provided through the Artful Learning professional development, the Artful Learning 
website, or through other targeted professional development. 

• More focused coaching and observation: A number of new and Artful Learning- 
experienced teachers reported that they would like to have more opportunities for 
immediate feedback and “critiques” and “observations” of their units. They 
expressed a desire for Artful Learning experts to visit and critique their Artful 
Learning unit and lessons to assure that they are implementing the program 
effectively and target areas for improvement. Logistically, the Artful Learning 
coaches are the individuals best poised to provide this sort of feedback. In order to 
facilitate the feedback/observation process, coaches could be provided with 
observation forms and a more formal structure for feedback to teachers. 
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Evaluation. As a competitive school reform model, evaluation will continue to be a 
eoneern for the Artful Learning program whether they maintain a formal external evaluator 
or not. Even without a formal external evaluation, the Artful Learning program will need to 
have internal evaluation struetures in place to continue to monitor the program’s quality and 
effectiveness. Following are some recommendations to consider in terms of future evaluation 
endeavors: 

• Student-level achievement data: An ultimate eoneern among educators and policy- 
makers is evidenee of school reform program impaet on student aehievement. As 
this report deseribes, there are some promising trends in the school-level data 
regarding the program schools in comparison to schools not using the program. 
However, individual student-level data, preferably from Artful Learning partieipants 
and non-participants, is needed in order to truly demonstrate program impact on 
student standardized test scores. Access to sueh data, however, will require not only 
district agreement, but also time and collaboration with district assessment offices. 
Such agreements and time should be negotiated with distriets up front when sehools 
first sign on for the Artful Learning program. 

• Other student indicators: The only student data available for the present evaluation 
was school-level achievement data. Artful Learning teaehers additionally highlight 
student engagement/interest as a primary area of impaet for the program. Future 
evaluation undertaking could thus include a student engagement measure, which 
could be administered to students prior to beginning Artful Learning instruetion and 
at 1-2 points later in the sehool year. Again, this undertaking would require the buy- 
in of Artful Learning participating schools and teachers. 

• Targeted case studies: The recommendations described above require a level of 
district commitment across multiple states that may not be possible for all Artful 
Learning sehools, especially in terms of attempts to secure a eomparison group of 
students. A feasible approaeh may be focusing on 2-3 Artful Learning sehools that 
will be receptive to evaluation activities for ease studies, rather than trying to collect 
information broadly from all Artful Learning participants. Optimally, this 
undertaking would not only require a sehooTs commitment to partieipate and share 
student level data (with, of eourse, distriet approval) but also their help in 
identifying possible comparison groups within the school (e.g., new implementing 
teaehers, teachers who have yet to seale up). Sueh targeted data collection would 
help further support any broader data eolleeted from all Artful Learning sehools as a 
whole. 



48 




REFERENCES 



Griffin, N. C., & Schwartz, L. (2002). Artful Learning Evaluation: Preliminary Progress 
Report. Los Angeles: University of California, National Center for Researeh on 
Evaluation, Standards, and Student Testing (CRESST). 

Griffin, N. C., Miyoshi, J., & Kao, J. (2003). Evaluation of the Grammy Foundation ’s Artful 
Learning program: Year 2. Eos Angeles: University of California, National Center for 
Research on Evaluation, Standards, and Student Testing (CRESST). 

Netemeyer, R. G., Bearden, W. O., & Sharma, S. (2003). Scaling procedures. Newbury Park, 
CA: Sage Publications. 

No Child Left Behind Act of 2001, Pub. L. No. 107-110, § 1 15 Stat. 1425 (2002). 

Northwest Regional Educational Laboratory. (June, 2004). Catalog of School Reform 
Models. Retrieved June 16, 2003, from 
http://www.nwrel.org/sopd/eatalog/about/criteria.shtml 

Tabaohnick, B. G., & Fidell, L. S. (2000). Computer-assisted researeh design and analysis. 
Boston: Allyn and Baoon. 



49 




APPENDIX A: SAMPLE PROTOCOLS 
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Grammy Project 

Evaluation of the Artful Learning Program 

Teacher Interview Protocol 
Returning AL Program Participant (Blue) 

2003-4 

Interviewer: READ VERBATIM TO INTERVIEWEE 

Thank you for agreeing to participate in this interview. The purpose of this interview is to gain an understanding 
of the implementation of the Artful Learning Program. This interview will take approximately 45-60 minutes. 

Your name will not be used in the final transcription, and any identifying information will be deleted from the 
final transcript. No one will read the transcript other than the members of the UCLA Evaluation Team, and all 
contents of this interview are confidential. 

You may choose not to answer a question, and/or choose to terminate the interview if you do not feel 
comfortable. Participation in this study is strictly voluntary. The decision not to participate will in no way affect 
your relationship with the Artful Learning program, the Grammy foundation, or with UCLA. 

If there are sensitive issues that you would like to discuss, but prefer for them not to be entered into the 
transcription, the interviewer will honor your request and that portion of your interview will not be transcribed. 
You may also ask for clarification at any time if you don’t understand a question, would like clarification, or 
would like the question repeated. 

Do you have any questions? 

Okay, I have several questions I would like to ask you to discuss. If you are ready, we will begin. 
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BACKGROUND QUESTIONS 

1 . Background questions: I would like to ask you a few background questions about your school and your 
experiences as a teacher. 

Probes: 

• How long have you been teaching? 

• How long have you taught at your current school? 

• What subjects/grades do you currently teach? 

• What credentials/certification do you hold and how long have you held them? 

• How long has your school been involved in the Artful Learning program? 

• How long have you personally been implementing the program in your classroom? 

• What other curricular programs do you use in your classroom? Are these programs school-wide, or 
unique to your classroom? 

PROGRAM IMPLEMENTATION 

2. What is the overall attitude at your school towards the Artful Learning Program? 

• How well does the program integrate with other curricular programs used at your school/in your 
classroom? 

• How was the implementation process different this year than in years past? 

• Do you see any added benefits to your continued implementation of this year compared to previous 
years? 

IMPACT: PROGRAM 

3. What is the role of masterworks in your Artful Learning-based instruction? 

Probes: 

• Were there any difficulties in your understanding of the masterwork concept and how it might 
apply to your classroom? In finding appropriate masterworks to use? 

• What specific masterworks did you incorporate into instruction? 

• How specifically did you use the masterworks in the classroom? 

4. Have you participated in the development of any Artful Learning instructional units? If so, please describe 
your experiences with the design and implementation process. 

Probes: 

• Who was involved? How many teachers participated? 

• How many units did you design? 

• What was the content of the units? 

• What was the design process like? 

• Were there any difficulties in the design process? If so, how did you address them? 

5. Describe how state or national standards were considered in developing your Bernstein Unit(s). 

Probes: 

• Were standards easier to consider for some of the program components than for others (Experience, 
Inquire, Create, and Reflect)? 

• If YES, which components were easier for considering standards and why? 

• Which components were difficult for considering standards and why? 

IMPACT: ASSESSMENT 

6. What impact, if any, did the Artful Learning model have on how you assess student progress? 

Probes: 

• Use of portfolio assessment strategies? If there was no impact, why? 

• Did you use portfolios before? If so, are there any differences in how you use them now compare to 
before the Artful Learning Program? 

• What was your implementation of the components of the portfolio system - process - repertoire - 
presentation? 
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PROFESSIONAL DEVELOPMENT 

7. What types of professional development activities did you participate in during the school year to support 
the implementation of the Artful Learning program? How useful to you were these activities? 

Probes: 

• Who offered the activities? 

• What level of training did you attend? (Level I, Level II, Level III?) 

• What did you leam that was useful to you? 

• How could the content or the process of the professional development be improved? 

8. Did any of your professional development experiences include curricular mapping? If so, 

• What is your understanding of the curricular mapping process? 

• Have you participated in any curricular mapping activities at your school? If so, please describe. 

Did you find these activities useful? Why/why not? 



SUPPORT 

9. Does your school have an Artful Learning Coach? What types of support does this person provide to 
you/your school? 

Probes: 

• Is the support provided useful? What additional support/assistance would you like to receive from 
your coach? 

10. Please describe the extent of support your school received from the Grammy Foundation in the 
implementation of the Artful Learning program (a lot, some, a little, none?). Were there areas you would 
have liked to receive more support in? 

Probes: 

• Additional professional development from the Grammy Foundation? 

• Technical support? Curricular support? Support for changes in school organization/structure? 

• Did you use the LBC website as a resource? What aspect(s), if any, of the website did you find 
useful, and how specifically did you use them? 

• 

1 1 . How would you describe the level of support for Artful Learning implementation that you received from 
your school principal? (a lot, some, a little, none?). What are your reasons for this description? 

Probes: 

• Were there any specific areas you would have liked to receive more support in? 

• Any specific examples of how your principal supported the implementation process? 

• Did your principal seem knowledgeable about the Artful Learning program? 



CAPACITY BUILDING AND SUSTAINABILITY 

12. What impact, if any, did the Artful Learning model have on your school’s organization? 
Probes: 

• Increased shared decision-making/leadership? 

• Increased communication/collaboration between teachers? 

• Increased communication between teachers and the principal? 

• School- wide data use and self-refiection/self-assessment? 



13. How has your participation in the Artful Learning program changed the way you interact with other 
teachers at your school? 

Probes: 

• How curriculum is designed? Nature of decision-making? 

• Any specific examples? 
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14. Has your school implemented Artful Learning Cadres? If so, what types of cadres have you participated 
in? 

Probes: 

• Describe specific examples of cadre activities. 

• Were the cadres useful at all to you in your implementation of the Artful Learning program? 

Why/ why not? 

15. Beyond the Artful Learning cadres, has your school developed any processes for group analysis of Artful 
Learning student work/rubrics? If so: 

• Describe the make up and procedures of this group(s). What types of material do you analyze? How 
do you analyze the materials as part of the group? How is this information used? 

• Please provide an example of this analysis process. 

16. If this is not your school’s first year participating in the Artful Learning Model, what other changes have 
you noticed this year in implementation compared to previous years? 

Probes: 

• Nature and quality of support and professional development from the Grammy Foundation? 

• Nature and quality of materials provided by the Grammy foundation? 

• Level of implementation of the program at your school? 

• Teacher and administrator buy-in? 

• Improved comfort with the instructional process? 

• Changes in school organization and decision-making processes? 

• Any other specific examples? 

IMPACT: OVERALL PROGRAM 

17. Overall, what impact do you feel the Artful Learning model has had on your classroom and your school? 

Probes: 

• Teachers? (practices, attitudes, school culture). 

• Students? (interest, participation, quality of work, achievement) 

• Administrators? (relationship with teachers, decision-making strategies) 

• Parents? 

• The community? 



THANK YOU FOR YOUR TIME AND PARTICIPATION! 
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Grammy Project 

Evaluation of the Artful Learning Program 

Leadership Interview Protocol 
Returning AL Program Participant (green) 

2003-4 

Interviewer: READ VERBATIM TO INTERVIEWEE 

Thank you for agreeing to participate in this interview. The purpose of this interview is to 
gain an understanding of the implementation of the Artful Learning Program. This interview 
will take approximately 45-60 minutes. 

Your name will not be used in the final transcription, and any identifying information will be 
deleted from the final transcript. No one will read the transcript other than the members of the 
UCLA Evaluation Team, and all contents of this interview are confidential. 

You may choose not to answer a question, and/or choose to terminate the interview if you do 
not feel comfortable. Participation in this study is strictly voluntary. The decision not to 
participate will in no way affect your relationship with the Artful Learning program, the 
Grammy foundation, or with UCLA. 

If there are sensitive issues that you would like to discuss, but prefer for them not to be 
entered into the transcription, the interviewer will honor your request and that portion of your 
interview will not be transcribed. You may also ask for clarification at any time if you don’t 
understand a question, would like clarification, or would like the question repeated. 

Do you have any questions? 

Okay, I have several questions I would like to ask you to discuss. If you are ready, we will 
begin. 
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School Type: Elementary School Middle School High School 

School #: 

The interviewee is: a coach/teacher a coach/administrator 

an administrator other 



BACKGROUND QUESTIONS 

1. Background questions: I would like to ask you a few background questions about your school and your 
experiences as an administrator (note: skip these questions if we already have this information). 

Probes: 

• How long have you been an administrator/teacher? 

• How long have you been an administrator/ AL coach at your current school? 

• How long has your school been involved in the Artful Learning program? 

PROGRAM IMPLEMENTATION 

2. How would you rate the level of implementation of the Artful Learning program at your school site? 
(Majority implementation? Full implementation? Partial implementation?) Ask teacher to define his/her 
rating of “majority,” “full,” or “partial” implementation. 

Do you think there are any changes in implementation level this year compared to last? 

Probes: 

• How many teachers have gone through the Artful Learning training? 

• How many teachers were trained this year? How many in years past? 

• What approach did your school site take in training teachers and in implementing the Artful 
Learning Program (e.g., whole school at once, phase-in teachers by grade, experience, etc.)? 

• Describe ways in which the program integrates other curricular programs used at your school in 
your teachers’ lessons. Any other examples? 

• What other curricular programs are used by teachers at your school (e.g.. Open Court, Houghton- 
Mifflin Mathematics, etc.)? 

• Are these programs school-wide, grade-wide, or unique to each teacher’s class? 

3. Describe your school site’s overall experience with implementing the Artful Learning curriculum in the 
classroom. 

Probes: 

• Discuss some of the barriers that emerged during the implementation process. 

• What are some of the reasons that teachers have experienced struggles during the implementation 
process? 

• Discuss some of the positive experiences that emerged during the implementation process. 

• What are some of the reasons why teachers may have experienced ease during the implementation 
process? 

• Have there been any differences in the ease or difficulty of Artful Learning implementation this 
year compared to years past? Please describe. What do you think led to these differences? 



58 




IMPACT: PROGRAM 

(Questions 4 and 5 for Artful Learning Coaches ONLY.) 



4. What is the role of masterworks in teachers Artful Learning-based instruction? 

Probes: 

• Were there any difficulties in their understanding of the masterwork concept and how it might 
apply to their classroom? In finding appropriate masterworks to use? 

• What types of masterworks did they incorporate into instruction? 

• How specifically did they use the masterworks in the classroom? 

5. I would like you to consider each of the components: EXPERIENCE — INQUIRE — CREATE — 
REFLECT. Can you provide examples of each of these components of the model? EXPERIENCE? 
INQUIRE? CREATE? REFLECT? 

Probes: 

• Was implementation any different this year than in years past? 

• (Elementary level ONLY) Were all components of the model incorporated across subject areas? 

• Were some components easier to implement than other? 

IMPACT: ASSESSMENT 

(Question 6 for Artful Learning Coaches ONLY.) 

6. Did the Artful Learning model impact the manner by which your teachers assess student progress? 
Probes: 

• Describe any pre-existing assessment methods that were modified by your teachers as a result of 
being a participant in the Artful Learning Program. 

• Describe any new assessment methods that were influenced and developed by your teachers as a 
result of being a participant in the Artful Learning Program. 

• Describe the use of Artful Learning Student Portfolio assessment strategies by your teachers. How 
many teachers use it? How long have they been using it? 

• Describe ways in which teachers have used existing or new assessment results to change or inform 
their classroom practice, as a result of participation in the Artful Learning Program. 

• How are standardized test results used at your school site to inform curriculum and practice? Has 
your participation in the Artful Learning program had any impact on how this is done? 

7. Are there any other differences this year compared to years past in your experiences with the 
implementation of the Artful Learning model that we have not asked you about? Please describe. 

PROFESSIONAL DEVELOPMENT 

8. Did you attend any professional development training during the summer or school year to support the 
implementation of the Artful Learning program? 

Probes: 

• Who offered the training? 

• Describe the content of the training. 

• How would you rate the training? Excellent — Fair — Poor 

• Can you cite some examples to support your rating of E/F/P? 

• Were there any improvements in your professional development from last year to this year? - 
Describe. 

• How could the professional development training activities be improved for 
administrators/coaches? 
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9. Did any of your professional development experiences cover curricular mapping? 

If YES, probes: 

• What is your understanding of curricular mapping (explain in detail)? 

• Is curricular mapping being coordinated at your school site? 

-Who is involved in curricular mapping? 

-How easy or difficult was it to develop curricular mapping? Describe. 

-How easy or difficult was it for teachers to grasp the concept and implement curricular mapping? 
Describe. 

SUPPORT 

10. How would you rate the level of support provided by your school district for the Artful Learning program 
at your school site? 

1 1 . Describe the support from your school district that your school site received during the implementation of 
the Artful Learning program (e.g., low, moderate, high). 

Probes: 

• Describe the benefits, if any, of the support provided by your school district as it relates to the 
Artful Learning Program and your school site. 

• Describe the barriers, if any, of the lack of support provided by your school district as it relates to 
the Artful Learning Program and your school site. 

• a) IF applicable. Describe any measures taken by you/your school site to initiate more district 
support, as this support relates to the Artful Learning Program. 

• b) IF applicable. Discuss ways that might help to facilitate more district support for your school 
site, as this support relates to the Artful Learning Program. 

• Have there been any changes in the support from the district for the Artful Program in this year 
compared to previous years? Please describe. 

12. Describe the support from the Grammy foundation that your school site received this year. 

Probes: 

• Support to principals? 

• Support to teachers? 

• Technical support? 

• Materials? 

• Are there (any other) areas you received or would have liked to receive more support in? 

• Are there any differences in the support you received this year compared to previous years ? Please 
describe. 

CAPACITY BUILDING (INCLUDING LEADERSHIP) AND SUSTAINABILITY 

13. What impact did the Artful Learning model have on your school’s organization in the following FOUR 
areas? 

• Communication between teachers? 

• Communication between teachers and the Artful Learning coaches? 

• Communication between teachers and the principal? 

• School-wide self-reflection and/or self-assessment? 

14. (RETURNING TEACHERS ONLY) Has your school implemented Artful Learning Cadres? If so, what 
types of cadres have you participated in? 

• Describe specific examples of cadre activities. 

• Were the cadres useful at all to you in your implementation of the Artful Learning program? 

Why/ why not? 
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15. What has been the impact of teacher turnover on the implementation of the Artful Learning program? 
Probes: 

• What has been the level of teacher turnover at your school since the implementation of the Artful 
Learning program? 

16. What types of support would be necessary to sustain the Artful Learning Program. 

• From within your school? 

• From your school district? 

• From the Grammy Foundation? 

IMPACT: OVERALL PROGRAM 

17. Describe the overall impact that the Artful Learning model has had on your school site. 

Probes: 

• Teachers 

• Students 

• Administrators 

• Parents 

• Community 

• Is there any impact that is new for your school this year? 



THANK YOU FOR YOUR TIME AND PARTICIPATION! 
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Artful Learning Level I Professional Development Training Pre-test Snrvey 
Session Date: Session Location: 

ID Number: 

Background information: 

Grade(s) you teach: Number of years using the Artful Learning model: 

Number of years teaching elementary school: Number of years teaching secondary school: 

Subjects taught (if applicable): 

Please rate your level of knowledge on the following topics: 





1 

(Novice) 


2 


3 


4 


5 

(Expert) 


1. 


Selecting appropriate works of art to (visual, musical, kinesthetic, literature) 
use in your curriculum 


1 


2 


3 


4 


5 


2. 


Techniques for using works of arts (visual, musical, kinesthetic, literature) as 
an educational tool throughout the curriculum 


1 


2 


3 


4 


5 


3. 


Developing questions for students to use in conducting inquiry/research as 
part of their regular classroom activities. 


1 


2 


3 


4 


5 


4. 


Assisting students in conducting inquiry/research as part of their regular 
classroom activities. 


1 


2 


3 


4 


5 


5. 


Teaching students to reflect on their own work as an educational tool 


1 


2 


3 


4 


5 


6. 


Designing lessons throughout the curriculum for students to produce their 
own creative work (e.g., art, music, literature, dance). 


1 


2 


3 


4 


5 


7. 


Supporting students in the creation of artistic works (music, dance, literature, 
visual art) as part of your regular classroom curriculum 


1 


2 


3 


4 


5 


8. 


Using self-reflection to examine and improve your own curriculum and 
instructional techniques. 


1 


2 


3 


4 


5 


9. 


Conducting action research in my classroom to assess your own instructional 
techniques or curriculum 


1 


2 


3 


4 


5 


10. 


Using reflective techniques at the school level (e.g., with other teachers and 
administrators) to examine and improve school-wide instructional strategies 


1 


2 


3 


4 


5 


11. 


Conducting action research at the school level to examine school-wide 
curriculum and instruction 


1 


2 


3 


4 


5 


12. 


Using reflective techniques at the school level (e.g., with other teachers and 
administrators) to examine school organizational strategies. 


1 


2 


3 


4 


5 


13. 


Using technology to support instruction throughout the curriculum 


1 


2 


3 


4 


5 


14. 


Using a portfolio assessment system 


1 


2 


3 


4 


5 


15. 


Using student assessment results to plan and refine your classroom practice 


1 


2 


3 


4 


5 


16. 


Using assessment results to evaluate your own effectiveness as a teacher 


1 


2 


3 


4 


5 


17. 


Using assessment results to help select instructional materials 


1 


2 


3 


4 


5 


18. 


Developing student literacy 


1 


2 


3 


4 


5 


19. 


Instructing English language learners 


1 


2 


3 


4 


5 


20. 


Instructing students in special education programs 


1 


2 


3 


4 


5 


21. 


Coordinating your curriculum with that of other teachers at my school 


1 


2 


3 


4 


5 


22. 


Coordinating your curriculum with state standards 


1 


2 


3 


4 


5 


23. 


Coordinating your curriculum with national standards 


1 


2 


3 


4 


5 
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Please indicate the extent to which you agree or disagree with the following statements: 





Strongly 

disagree 

1 


Disagree 

2 


Disagree 
slightly 
more than 
agree 
3 


Agree 
slightly 
more than 
disagree 
4 


Agree 

5 


Strongly 

Agree 

6 


Don’t 

Know/ 

Unsure 

7 


24. 1 feel that there is a need for the 
Artful Learning program at my 
school. 


1 


2 


3 


4 


5 


6 


7 


25. 1 feel that the Artful Learning 
program will be useful to me as 
part of my day-to-day classroom 
practices. 


1 


2 


3 


4 


5 


6 


7 


26. 1 will be able to integrate the 

Artful Learning program with the 
other curricular programs/ 
materials 1 already use in my 
classroom. 


1 


2 


3 


4 


5 


6 


7 


27. There is support for 

implementation of the Artful 
Learning program from the 
administrators at my school. 


1 


2 


3 


4 


5 


6 


7 


28. There is support for 

implementation of the Artful 
Learning program from most of 
the teachers at my school. 


1 


2 


3 


4 


5 


6 


7 


29. 1 am looking forward to 

implementing the Artful Learning 
program in my classroom this fall. 


1 


2 


3 


4 


5 


6 


7 
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Artful Learning Level I Professional Development Session Post-test Snrvey 



Session Date: Session Location: 

ID Number: 



1. Please rate your level of knowledge on the following topics: 





1 

(Novice) 


2 


3 


4 


5 

(Expert) 


1. Selecting appropriate works of art to (visual, musical, kinesthetic, 
literature) use in your curriculum 


1 


2 


3 


4 


5 


2. Techniques for using works of arts (visual, musical, kinesthetic, 
literature) as an educational tool throughout the curriculum 


1 


2 


3 


4 


5 


3. Developing questions for students to use in conducting inquiry/research 
as part of their regular classroom activities. 


1 


2 


3 


4 


5 


4. Assisting students in conducting inquiry/research as part of their regular 
classroom activities. 


1 


2 


3 


4 


5 


5. Teaching students to reflect on their own work as an educational tool 


1 


2 


3 


4 


5 


6. Designing lessons throughout the curriculum for students to produce 
their own creative work (e.g., art, music, literature, dance). 


1 


2 


3 


4 


5 


7. Supporting students in the creation of artistic works (music, dance, 
literature, visual art) as part of your regular classroom curriculum 


1 


2 


3 


4 


5 


8. Using self-reflection to examine and improve my own curriculum and 
instructional techniques. 


1 


2 


3 


4 


5 


9. Conducting action research in my classroom to assess my own 
instructional techniques or curriculum 


1 


2 


3 


4 


5 


10. Using reflective techniques at the school level (e.g., with other teachers 
and administrators) to examine and improve school-wide instructional 
strategies 


1 


2 


3 


4 


5 


1 1 . Conducting action research at the school level to examine school-wide 
curriculum and instruction 


1 


2 


3 


4 


5 


12. Using reflective techniques at the school level (e.g., with other teachers 
and administrators) to examine school organizational strategies. 


1 


2 


3 


4 


5 


13. Using technology to support instruction throughout the curriculum 


1 


2 


3 


4 


5 


14. Using a portfolio assessment system 


1 


2 


3 


4 


5 


15. Using student assessment results to plan and refine my classroom 
practice 


1 


2 


3 


4 


5 


16. Using assessment results to evaluate my own effectiveness as a teacher 


1 


2 


3 


4 


5 


17. Using assessment results to help select instructional materials 


1 


2 


3 


4 


5 


18. Developing student literacy 


1 


2 


3 


4 


5 


19. Instructing English language learners 


1 


2 


3 


4 


5 


20. Instructing students in special education programs 


1 


2 


3 


4 


5 


2 1 . Coordinate my curriculum with that of other teachers at my school 


1 


2 


3 


4 


5 


22. Coordinating my curriculum with district and state standards 


1 


2 


3 


4 


5 


23. Coordinating my curriculum with national standards 


1 


2 


3 


4 


5 
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2. Please indicate the extent to which you agree with the following statements about this Artful Learning training session: 





Strongly 

disagree 

1 


Disagree 

2 


Disagree 
slightly more 
than agree 

3 


Agree slightly 
more than 
disagree 
4 


Agree 

5 


Strongly 

Agree 

6 


1 . Overall, the training provided me with 
knowledge or skills that are very useful 
in my classroom practices 


1 


2 


3 


4 


5 


6 


2. The training facilitators regularly made 
links between the content covered and 
specific instructional practices 


1 


2 


3 


4 


5 


6 


3. The training sessions that I attended 
were well connected to each other 


1 


2 


3 


4 


5 


6 



3. How useful to you were the following aspects of this Artful Learning training session: 





Not at all 
useful 
1 


Slightly 

useful 

2 


Somewhat 

useful 

3 


Useful 

4 


Very 

Useful 

5 


N/A 

I did not have 
the opportunity 
to do this 
6 


1 . Practicing the instructional strategies 
that were discussed 


1 


2 


3 


4 


5 


6 


2. Discussing/reflecting with other 

teachers about what you had learned 


1 


2 


3 


4 


5 


6 


3. Getting feedback from trainers about 
your teaching practices 


1 


2 


3 


4 


5 


6 



4. How well do you feel that this training session prepared you to undertake the following activities: 





Not at all 
1 


Slightly 

2 


Somewhat 

3 


Adequately 

4 


More than 
adequately 
5 


1 . Implementing the Artful Learning instructional 
strategies in my classroom 


1 


2 


3 


4 


5 


2. Implementing the Artful Learning Portfolio 
assessment system 


1 


2 


3 


4 


5 


3. Implementing the school organizational aspects 
of the Artful Learning model 


1 


2 


3 


4 


5 



5. Please indicate the extent to which you agree or disagree with the following statements. 





Strongly 

disagree 

1 


Disagree 

2 


Disagree 
slightly 
more than 
agree 

3 


Agree 
slightly 
more than 
disagree 
4 


Agree 

5 


Strongly 

Agree 

6 


Don’t 

Know/ 

Unsure 

7 


1 . I feel that there is a need for the Artful 
Learning program at my school. 


1 


2 


3 


4 


5 


6 


7 


2. I feel that the Artful Learning program 
will be useful to me as part of my day- 
to-day classroom practices. 


1 


2 


3 


4 


5 


6 


7 



( Question continues on next page) 
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Strongly 

disagree 

1 


Disagree 

2 


Disagree 
slightly 
more than 
agree 
3 


Agree 
slightly 
more than 
disagree 
4 


Agree 

5 


Strongly 

Agree 

6 


Don’t 

Know/ 

Unsure 

7 


3. I will be able to integrate the Artful 
Learning program with the other 
curricular programs/ materials I already 
use in my classroom. 


1 


2 


3 


4 


5 


6 


7 


4. There is support for implementation of 
the Artful Learning program from the 
administrators at my school. 


1 


2 


3 


4 


5 


6 


7 


5. There is support for implementation of 
the Artful Learning program from most 
of the teachers at my school. 


1 


2 


3 


4 


5 


6 


7 


6. I am looking forward to implementing 
the Artful Learning program in my 
classroom this fall. 


1 


2 


3 


4 


5 


6 


7 



6. What activity helped the most? Why? 



7. What activity helped the least? Why? 



8. What are some of the current/most important obstacles to using what you’ve learned? 



9. 


Please indicate names of facilitators: 


#1 






#2_ 




10. 


#1 facilitator’s knowledge: 


(low) 1 


2 


3 


4 


5 (high) 




#1 facilitator’s ability to present: 


(low) 1 


2 


3 


4 


5 (high) 


11. 


Suggestions you might have for facilitator #1 : 










12. 


#2 facilitator’s knowledge: 


(low) 1 


2 


3 


4 


5 (high) 




#2 facilitator’s ability to present: 


(low) 1 


2 


3 


4 


5 (high) 



13. 



Suggestions you might have for facilitator # 2: 



14. The materials: (low quality) 1 2 3 4 5 (high quality) 

15. The facility and the arrangements: 1 2 3 4 5 

16. As you use the Artful Learning Model, how is it most likely to help you, your students and your school? 



17. Other comments or suggestions to improve the Bernstein Artful Learning Model School Improvement 
professional development: 
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Artful Learning Level II Professional Development Training Pre-test Survey 
Session Date: Session Location: 

ID Number: 

Background information: 

Grade(s) you teach: Number of years using the Artful Learning model: 

Number of years teaching elementary school: Number of years teaching secondary school: 

Subjects taught (if applicable): 

Please rate your level of knowledge on the following topics: 





1 

(Novice) 


2 


3 


4 


5 

(Expert) 


1. 


Selecting appropriate masterworks (visual, musical, kinesthetic) to use in your 
curriculum 


1 


2 


3 


4 


5 


2. 


Instructional techniques for using masterworks in your classroom 


1 


2 


3 


4 


5 


3. 


Designing Artful Learning Instructional Units 


1 


2 


3 


4 


5 


4. 


Designing the Experience component of your Artful Learning Units 


1 


2 


3 


4 


5 


5. 


Designing the Inquire component of your Artful Learning Units 


1 


2 


3 


4 


5 


6. 


Designing the Create component of your Artful Learning Units 


1 


2 


3 


4 


5 


7. 


Designing the Reflect component of your Artful Learning Units 


1 


2 


3 


4 


5 


8. 


Developing significant question for your Artful Learning Units 


1 


2 


3 


4 


5 


9. 


Selecting a concept for your Artful Learning Unit(s) 


1 


2 


3 


4 


5 


10. 


Linking your Artful Learning Units to state and local standards 


1 


2 


3 


4 


5 


11. 


Designing a Unit that alters the Artful Learning sequence (Experience, 
Inquire, Create, Reflect) 


1 


2 


3 


4 


5 


12. 


Using curricular mapping techniques to plan instruction 


1 


2 


3 


4 


5 


13. 


Using self-reflection to examine and improve your own curriculum and 
instructional techniques 


1 


2 


3 


4 


5 


14. 


Conducting action research in your classroom to assess your own instructional 
techniques or curriculum 


1 


2 


3 


4 


5 


15. 


Meeting as a team with the other Artful Learning teachers at your school to 
support program implementation 


1 


2 


3 


4 


5 


16. 


Meeting as a team with the other Artful Learning teachers at your school to 
design Artful Learning units 


1 


2 


3 


4 


5 


17. 


Meeting as a team with the other Artful Learning teachers at your school to 
conduct action research at the school level 


1 


2 


3 


4 


5 


18. 


Using technology to support your classroom instruction 


1 


2 


3 


4 


5 


19. 


Using technology resources to help design your Artful Learning Units 


1 


2 


3 


4 


5 


20. 


Using a portfolio assessment system 


1 


2 


3 


4 


5 


21. 


Assessing students during the Experience component of your Artful Learning 
Units 


1 


2 


3 


4 


5 


22. 


Assessing students during the Inquire component of your Artful Learning 
Units 


1 


2 


3 


4 


5 


23. 


Assessing students during the Create component of your Artful Learning Units 


1 


2 


3 


4 


5 
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1 

(Novice) 


2 


3 


4 


5 

(Expert) 


24. Assessing students during the Reflect component of your Artful Learning 
Units 


1 


2 


3 


4 


5 


25. Coordinating your curriculum with that of other teachers at your school 


1 


2 


3 


4 


5 


26. Coordinating your curriculum with district and state standards 


1 


2 


3 


4 


5 


27. Coordinating your curriculum with national standards 


1 


2 


3 


4 


5 


28. Developing a professional portfolio of your work to chart your professional 
growth 


1 


2 


3 


4 


5 


29. Using a professional portfolio of your work to present information about your 
instructional skills to others 


1 


2 


3 


4 


5 



Please indicate the extent to which you agree or disagree with the following statements. 





Strongly 

disagree 

1 


Disagree 

2 


Disagree 
slightly 
more than 
agree 

3 


Agree 
slightly 
more than 
disagree 
4 


Agree 

5 


Strongly 

Agree 

6 


Don’t 

Know/ 

Unsure 

7 


1 . I feel that there is a need for the 
Artful Learning program at my 
school. 


1 


2 


3 


4 


5 


6 


7 


2. I feel that the Artful Learning 
program is useful to me as part of 
my day-to-day classroom practices. 


1 


2 


3 


4 


5 


6 


7 


3. I will be able to integrate the Artful 
Learning program with the other 
curricular programs/ materials I 
already use in my classroom. 


1 


2 


3 


4 


5 


6 


7 


4. There is support for 

implementation of the Artful 
Learning program from the 
administrators at my school. 


1 


2 


3 


4 


5 


6 


7 


5. There is support for 

implementation of the Artful 
Learning program from most of the 
teachers at my school. 


1 


2 


3 


4 


5 


6 


7 


6. I am looking forward continuing to 
use the Artful Learning program in 
my classroom. 


1 


2 


3 


4 


5 


6 


7 
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Artful Learning Level II Professional Development Session Post-test Survey 



Session Date: Session Loeation: 

ID Number: 

1. Please rate your level of knowledge on the following topics: 





(Novice) 

1 


2 


3 


4 


(Expert) 

5 


1. 


Selecting appropriate masterworks (visual, musical, kinesthetic) to use in your 
curriculum 


1 


2 


3 


4 


5 


2. 


Instructional techniques for using masterworks in your classroom 


1 


2 


3 


4 


5 


3. 


Designing Artful Learning Instructional Units 


1 


2 


3 


4 


5 


4. 


Designing the Experience component of your Artful Learning Units 


1 


2 


3 


4 


5 


5. 


Designing the Inquire component of your Artful Learning Units 


1 


2 


3 


4 


5 


6. 


Designing the Create component of your Artful Learning Units 


1 


2 


3 


4 


5 


7. 


Designing the Reflect component of your Artful Learning Units 


1 


2 


3 


4 


5 


8. 


Developing significant question for your Artful Learning Units 


1 


2 


3 


4 


5 


9. 


Selecting a concept for your Artful Learning Unit(s) 


1 


2 


3 


4 


5 


10. 


Linking your Artful Learning Units to state and local standards 


1 


2 


3 


4 


5 


11. 


Designing a Unit that alters the Artful Learning sequence (Experience, Inquire, 
Create, Reflect) 


1 


2 


3 


4 


5 


12. 


Using curricular mapping techniques to plan instruction 


1 


2 


3 


4 


5 


13. 


Using self-reflection to examine and improve your own curriculum and 
instructional techniques 


1 


2 


3 


4 


5 


14. 


Conducting action research in your classroom to assess your own instructional 
techniques or curriculum 


1 


2 


3 


4 


5 


15. 


Meeting as a team with the other Artful Learning teachers at your school to 
support program implementation 


1 


2 


3 


4 


5 


16. 


Meeting as a team with the other Artful Learning teachers at your school to 
design Artful Learning units 


1 


2 


3 


4 


5 


17. 


Meeting as a team with the other Artful Learning teachers at your school to 
conduct action research at the school level 


1 


2 


3 


4 


5 


18. 


Using technology to support your classroom instruction 


1 


2 


3 


4 


5 


19. 


Using technology resources to help design your Artful Learning Units 


1 


2 


3 


4 


5 


20. 


Using a portfolio assessment system 


1 


2 


3 


4 


5 


21. 


Assessing students during the Experience component of your Artful Learning 
Units 


1 


2 


3 


4 


5 


22. 


Assessing students during the Inquire component of your Artful Learning Units 


1 


2 


3 


4 


5 


23. 


Assessing students during the Create component of your Artful Learning Units 


1 


2 


3 


4 


5 


24. 


Assessing students during the Reflect component of your Artful Learning Units 


1 


2 


3 


4 


5 


25. 


Coordinating your curriculum with that of other teachers at your school 


1 


2 


3 


4 


5 


26. 


Coordinating your curriculum with district and state standards 


1 


2 


3 


4 


5 


27. 


Coordinating your curriculum with national standards 


1 


2 


3 


4 


5 


28. 


Developing a professional portfolio of your work to chart your professional 
growth 


1 


2 


3 


4 


5 


29. 


Using a professional portfolio of your work to present information about your 
instructional skills to others 


1 


2 


3 


4 


5 
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2. Please indicate the extent to which you agree with the following statements about this Artful Learning training session: 





Strongly 

disagree 

1 


Disagree 

2 


Disagree 
slightly 
more than 
agree 
3 


Agree 
slightly 
more than 
disagree 
4 


Agree 

5 


Strongly 

Agree 

6 


1 . Overall, the training provided me with 

knowledge or skills that are very useful in my 
classroom practices 


1 


2 


3 


4 


5 


6 


2. Overall, the training provided me with 

knowledge or skills that will help me to better 
collaborate with other teachers at my school. 


1 


2 


3 


4 


5 


6 


3. The training facilitators regularly made links 
between the content covered and specific 
instructional practices 


1 


2 


3 


4 


5 


6 


4. This session was well-connected to previous 
Artful Learning professional development I 
have attended 


1 


2 


3 


4 


5 


6 



3. How useful to you were the following aspects of this Artful Learning training session: 







Not at all 
useful 


Slightly 

useful 


Somewhat 

useful 


Useful 


Very 

Useful 


N/A I did 
not have the 
opportunity 
to do this 






1 


2 


3 


4 


5 


6 


1. 


Practicing the instructional strategies that 
were discussed 


1 


2 


3 


4 


5 


6 


2. 


Completing worksheet/workbook activities 


1 


2 


3 


4 


5 


6 


3. 


Discussing/reflecting with other teachers 
about what you had learned 


1 


2 


3 


4 


5 


6 


4. 


Getting feedback from trainers about your 
teaching practices 


1 


2 


3 


4 


5 


6 



4. How well do you feel that this training session prepared you to undertake the following activities: 





Not at all 
1 


Slightly 

2 


Somewhat 

3 


Adequately 

4 


More than 
adequately 
5 


1. 


Designing Artful Learning Units 


1 


2 


3 


4 


5 


2. 


Implementing the Artful Learning instructional 
strategies in my classroom 


1 


2 


3 


4 


5 


3. 


Implementing the Artful Learning Portfolio 
assessment system 


1 


2 


3 


4 


5 


4. 


Implementing Artful Learning cadres at my 
school 


1 


2 


3 


4 


5 


5. 


Developing my professional portfolio 


1 


2 


3 


4 


5 
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5. Please indicate the extent to which you agree or disagree with the following statements. 





Strongly 

disagree 

1 


Disagree 

2 


Disagree 
slightly 
more than 
agree 
3 


Agree 
slightly 
more than 
disagree 
4 


Agree 

5 


Strongly 

Agree 

6 


Don’t 

Know/ 

Unsure 

7 


1 . There is a need for the Artful 
Learning program at my 
school. 


1 


2 


3 


4 


5 


6 


7 


2. The Artful Learning program 
is useful in my day-to-day 
classroom practices. 


1 


2 


3 


4 


5 


6 


7 


3. The information and 

techniques I learned in this 
session will help me to better 
implement the Artful Learning 
program in my classroom. 


1 


2 


3 


4 


5 


6 


7 


4. There is support for 

implementation of the Artful 
Learning program from the 
administrators at my school. 


1 


2 


3 


4 


5 


6 


7 


5. There is support for 

implementation of the Artful 
Learning program from most 
of the teachers at my school. 


1 


2 


3 


4 


5 


6 


7 


6. The information and 

techniques I learned in this 
session will help me to better 
collaborate with other teachers 
at my school. 


1 


2 


3 


4 


5 


6 


7 


7. I am looking forward to 
continuing to use the Artful 
Learning program in my 
classroom. 


1 


2 


3 


4 


5 


6 


7 



6. What activity helped the most? Why? 



7. What activity helped the least? Why? 



8. What are some of the current/most important obstacles to using what you’ve learned? 



9. 

10 . 

11 . 



Please indicate names of facilitators: #1 

#1 facilitator’s knowledge: (low) 1 2 

#1 facilitator’s ability to present: (low) 1 2 

Suggestions you might have for facilitator # 1 : 



#2 

3 4 5 (high) 

3 4 5 (high) 
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12. 


#2 facilitator’s knowledge: (low) 1 


2 


3 


4 


5 (high) 




#2 facilitator’s ability to present: (low) 1 


2 


3 


4 


5 (high) 


13. 


Suggestions you might have for facilitator # 2: 










14. 


The materials: (low quality) 1 


2 


3 


4 


5 (high quality) 


15. 


The facility and the arrangements: 1 


2 


3 


4 


5 



16. In your experience using the Artful Learning Model, how has it most helped you, your students and your 
school? 



17. Other comments or suggestions to improve the Bernstein Artful Learning Model School Improvement 
professional development: 
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ARTFUL LEARNING PROGRAM/LEONARD BERNSTEIN CENTER: 
IMPLEMENTA TION SURVEY 



Please answer the following questions about your typical experiences at your school during the present (2002-2003) school 
year. 

Please Shade Circles Like This A 0 

Not Like This -> ^ 0^ 



1 . What is your current position? 

O Classroom Teacher O Administrator O Resource Specialist O Other 



2. What category does the grade level(s) you teach fall in? 

O Kindergarten 
O Grade 1, 2, or 3 
O Grade 4 or 5 
O Grade 6, 7, or 8 
O Grade 9, 10, 11, or 12 
O N/A or None of the above 



3. How long have you been teaching? 

O Less that 1 year 
O 1-2 years 
O 3-5 years 
O 6-10 years 
O 1 1-15 years 
O More than 1 5 years 



4. When did you first implement the Artful Learning Program in your classroom? 

O This school year (2002 - 2003) 

O Last school year (2001 - 2002) 

O Prior to the 2001 - 2002 school year 
O I have not implemented it in my classroom yet 



5. How many Artful Learning Professional Development trainings have you attended? 



0 


1 


2 


3 


4 


More than 4 


o 


o 


o 


O 


O 


O 



6. To what extent do you agree or disagree with the following statements: 





Strongly 

disagree 


Disagree 


Disagree 
slightly 
more than 
agree 


Agree 
slightly 
more than 
disagree 


Agree 


Strongly 

agree 


Not 

applicable 


a. When planning a lesson, I 
take state and national 
standards into account. 


O 


O 


O 


O 


O 


o 


O 


b. I incorporate these standards 
in my classroom teaching. 


O 


O 


O 


O 


o 


o 


O 


c. My students are aware of the 
standards. 


o 


o 


o 


o 


o 


o 


o 


d. Most of my students have the 
ability to meet the standards. 


o 


o 


o 


o 


o 


o 


o 


e. I can teach the standards to a 
diverse group of learners. 


o 


o 


o 


o 


o 


o 


o 
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7. How familiar are you with the following? 





Very 

familiar 


Somewhat 

familiar 


Familiar 


Not very 
familiar 


Not 

familiar at 
all 


Not 

applicable 


a. State/national standards at your 
grade level. 


O 


O 


O 


O 


O 


O 


b. State/national standards at the grade 
level below. 


O 


O 


O 


O 


O 


O 


c. State/national standards at the grade 
level above. 


o 


o 


o 


o 


o 


o 



8. How prepared do you feel to implement your curriculum in the following content areas? 





Not at all 


Slightly 


Somewhat 


Adequately 


More than 
adequately 


I do not 
teach this 
content 
area 


a. 


Math 


O 


O 


O 


O 


O 


O 


b. 


Science 


O 


o 


o 


O 


O 


O 


c. 


Social Studies 


o 


o 


o 


o 


o 


o 


d. 


Language Arts 


o 


o 


o 


o 


o 


o 


e. 


Art 


o 


o 


o 


o 


o 


o 


f. 


Music 


o 


o 


o 


o 


o 


o 



9. Based on the following scale: 





More than 
once a 
week 


Weekly 


Bi-weekly 


Monthly 


Never 


a. How often do you send information about your 
classroom home to parents? 


O 


O 


O 


O 


O 


b. How often do you meet with parents at school? 


O 


O 


O 


O 


O 



10. Based on the following scale: 





Never 


Once/twice 
a year 


3-5 times 
a year 


6-1 1 times 
a year 


12 or more 
times 
a year 


a. How often do you review achievement data for the 
students in your classroom? 


O 


o 


o 


o 


O 


b. How often do teachers and administrators formally 
meet together at your school? 


O 


o 


o 


o 


O 
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1 1 . When teachers meet at your school, approximately what percentage of time is typicallyspent on the following things? 
Round to the nearest percent. 





0% 


25% 


50% 


75% 


100% 


a. 


Sharing problems, stories, or concerns 


O 


o 


o 


o 


o 


b. 


Discussing the latest research and its 
implications for practice. 


O 


o 


o 


o 


o 


c. 


Analyzing student work. 


o 


o 


o 


o 


o 


d. 


Modifying the curriculum. 


o 


o 


o 


o 


o 


e. 


Discussing student achievement data. 


o 


o 


o 


o 


o 


f. 


Discussing school-wide goals. 


o 


o 


o 


o 


o 



12. Based on the following scale: 





Never 


Rarely 


Sometimes 


Often 


Very often 


Always 


a. 


How often do you change a lesson based on 
reviewing informal assessment results? 


O 


O 


O 


O 


O 


O 


b. 


How often do you change a lesson based on 
reviewing formal assessment results? 


O 


o 


o 


o 


O 


o 


13. 


How often do the following activities occur in a typical day in your classroom? 










Never 


Rarely 


Sometimes 


Often 


Very often 


Always 


a. 


Students are working together in small 
groups. 


O 


O 


O 


O 


O 


O 


b. 


Students are engaged in a whole class 
discussion. 


O 


O 


O 


O 


O 


O 


c. 


Students are reviewing and reflecting on 
their work with other classmates. 


o 


o 


o 


O 


o 


o 


d. 


Teacher is reviewing and reflecting on work 
with students. 


o 


o 


o 


O 


o 


o 


e. 


Student work samples are on display. 


o 


o 


o 


O 


o 


o 


f. 


Students are motivated. 


o 


o 


o 


O 


o 


o 
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14. To what extent do you agree or disagree with the following statement: 





Strongly 

disagree 


Disagree 


Disagree 
slightly 
more than 
agree 


Agree 
slightly 
more than 
disagree 


Agree 


Strongly 

agree 


Not applicable 
(I have not established 
specific behavior standards 
for my students) 


a. Students follow the 
behavior standards I 
have established in 
the classroom. 


O 


O 


O 


O 


O 


o 


O 



15. Approximately, what percentage of students at your school: 





0% 


25% 


50% 


75% 


100% 


a. participate in after school academic 
programs (homework clubs, tutoring, 
reading groups, computer classes)? 


O 


O 


O 


O 


o 


b. participate in after school extracurricular 
activities offered at your school (music, 
sports)? 


O 


O 


o 


o 


o 



16. How often do... 







Never 


Rarely 


Sometimes 


Often 


Very often 


Always 


a. 


you use tests with open-ended questions? 


O 


O 


O 


O 


O 


O 


b. 


you use extended writing assignments to 
evaluate student work? 


O 


o 


o 


o 


o 


o 


c. 


you use projects to evaluate student work? 


o 


o 


o 


o 


o 


o 


d. 


you use presentations to evaluate student 
work? 


o 


o 


o 


o 


o 


o 


e. 


you use a rubric in your assessments? 


o 


o 


o 


o 


o 


o 


f 


all teachers at the same grade level use the 
same assessments? 


o 


o 


o 


o 


o 


o 


g- 


you provide models of quality work to 
parents when giving an assignment? 


o 


o 


o 


o 


o 


o 



76 







17. To what extent do you agree or disagree with the following statements: 





Strongly 

disagree 


Disagree 


Disagree 
slightly 
more than 
agree 


Agree 
slightly 
more than 
disagree 


Agree 


Strongly 

agree 


Not 

applicable 


a. I understand the purpose of 
each assessment I use in the 
classroom. 


O 


O 


O 


O 


O 


o 


O 


b. I use the assessments to help 
modify my classroom 
practices. 


O 


O 


O 


O 


O 


o 


O 


c. I understand the purpose of 
state and/or district- wide 
standardized tests that my 
students complete. 


o 


O 


o 


o 


o 


o 


o 



18. Please estimate how often the following occur: 





Never 


Rarely 


Sometimes 


Often 


Very often 


Always 


a. Teachers’ input and opinions 
influence decision making at my 
school. 


O 


O 


O 


O 


O 


O 


b. Parents’ input and opinions influence 
decision making at my school. 


O 


O 


O 


O 


O 


o 


c. Paraprofessionals’ input and opinions 
influence decision making at my 
school. 


o 


o 


o 


O 


o 


o 


d. Teachers at my school are rewarded 
for their accomplishments. 


o 


o 


o 


O 


o 


o 



19. What means does your school use to share information with parents about school goals and mission? 





Never 


Rarely 


Sometimes 


Often 


Very often 


Always 


a. 


Meetings held more than monthly. 


O 


O 


O 


O 


O 


O 


b. 


Monthly meetings. 


O 


O 


O 


o 


O 


o 


c. 


N e wsletter/handouts . 


o 


o 


o 


o 


o 


o 
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20. To what extent do you agree or disagree with the following statements: 





Strongly 

disagree 


Disagree 


Disagree 
slightly 
more than 
agree 


Agree 
slightly 
more than 
disagree 


Agree 


Strongly 

agree 


Not 

applicable 


a. Classroom materials and supplies 
are readily available to me. 


O 


O 


O 


O 


O 


o 


O 


b. My school regularly monitors 
changes in student achievement. 


O 


O 


O 


O 


o 


o 


O 


c. My students maintain a portfolio of 
their assessments to track their 
progress. 


o 


O 


o 


o 


o 


o 


o 


d. I have access to student data on 
other schools with similar 
demographics. 


o 


O 


o 


o 


o 


o 


o 



21. To what extent do you agree or disagree with the following statements: 





Strongly 

disagree 


Disagree 


Disagree 
slightly 
more than 
agree 


Agree 
slightly 
more than 
disagree 


Agree 


Strongly 

agree 


Not 

applicable 
(We don’t 
have a 
mission 
statement) 


a. There is a shared vision for 
instruction among teachers and 
administrators at my school. 


O 


O 


O 


O 


O 


o 


O 


b. Most teachers at my school 
know the school mission. 


O 


O 


O 


O 


o 


o 


O 


c. Our mission statement 

addresses the future direction 
of the school. 


o 


O 


o 


o 


o 


o 


o 


d. Our mission statement 

emphasizes student learning. 


o 


O 


o 


o 


o 


o 


o 


e. Our mission addresses 
instructional methods. 


o 


O 


o 


o 


o 


o 


o 


f. Our mission addresses school 
organization. 


o 


O 


o 


o 


o 


o 


o 



22. Based on the following scale: 





Daily 


At least twice 
a week 


Weekly 


Bi-weekly 


Monthly 


a. Typically, how often do you add a 
student portfolio? 


O 


O 


O 


O 


O 


b. Typically, how often do you review a 
portfolio? 


O 


O 


O 


O 


O 
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23. Based on the following scale: 





None 


Less than 
1/2 hour 


1/2 hour 


1 hour 


2 hours 


More 
than 
2 hours 


a. During a typical school day, how much time 
is set aside by the school for teacher 
planning? 


O 


O 


O 


O 


O 


O 


b. During a typical week, how much time are 
you given for teacher team meetings? 


O 


O 


O 


O 


O 


O 



24. To what extent do you agree or disagree with the following statements about your school: 





Strongly 

disagree 


Disagree 


Disagree 
slightly 
more than 
agree 


Agree 
slightly 
more than 
disagree 


Agree 


Strongly 

agree 


Not 

applicable 


a. Adequate space is available to 
teachers for team meetings. 


O 


O 


O 


O 


O 


o 


O 


b. The school calendar includes 
days for teachers to work in 
teams (pupil free or substitute 
days). 


O 


O 


O 


O 


o 


o 


O 


c. When there is a conflict or 
disagreement amongst staff, it 
usually gets resolved fairly. 


o 


o 


o 


o 


o 


o 


o 


d. Teachers take responsibility for 
other teachers’ students when 
they are on the yard. 


o 


o 


o 


o 


o 


o 


o 


e. There is a sense of caring and 
support in the school 
community. 


o 


o 


o 


o 


o 


o 


o 
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25. To what extent do you agree or disagree with the following statements? 





Strongly 

disagree 


Disagree 


Disagree 
slightly 
more than 
agree 


Agree 
slightly 
more than 
disagree 


Agree 


Strongly 

agree 


Not 

applicable 


a. My school uses parents as 
resources in the classroom. 


O 


O 


O 


O 


O 


o 


O 


b. My school collaborates with 
other schools in my district. 


O 


O 


O 


O 


o 


o 


O 


c. My school uses professors or 
students from nearby 
universities as resources. 


o 


O 


o 


o 


o 


o 


o 


d. My school offers adequate 
opportunities for teacher 
professional development. 


o 


O 


o 


o 


o 


o 


o 


e. The professional development 
offered is aligned with the 
school mission/goals. 


o 


O 


o 


o 


o 


o 


o 


f Teachers at my school have a 
significant impact on decisions 
regarding the content of 
professional development. 


o 


O 


o 


o 


o 


o 


o 


g. The professional development 
activities offered at my school 
are useful to me in changing 
my classroom practices. 


o 


O 


o 


o 


o 


o 


o 


h. The professional development 
activities offered at my school 
provide research-based 
information. 


o 


O 


o 


o 


o 


o 


o 



26. How well did the Artful Learning (Leonard Bernstein Center) professional development activities you participated in this year 
prepare you for the implementations of Artful Learning Units? 



Not at all 


Slightly 


Somewhat 


Adequately 


More than 
adequately 


Not Applicable 
(I did not participate 
in the Artful Learning 
Professional 
Development 
Activities) 


O 


O 


O 


O 


O 


O 
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27. To what extent do you agree or disagree with the following statements about your school/classroom? 





Strongly 

disagree 


Disagree 


Disagree 
slightly 
more than 
agree 


Agree 
slightly 
more than 
disagree 


Agree 


Strongly 

agree 


Not 

applicable 


a. There is sufficient 

professional development 
at my school on 
understanding how to 
effectively implement 
school improvement. 


O 


O 


O 


O 


O 


o 


O 


b. My school maintains its 
commitment to 
improvement all year long. 


O 


O 


O 


O 


o 


o 


O 


c. There is a strong arts 
instruction program in 
place at my school. 


o 


o 


o 


o 


o 


o 


o 


d. At my school, arts 

instruction is incorporated 
into instruction in other 
subject areas. 


o 


o 


o 


o 


o 


o 


o 


e. When planning an Artful 
Learning Unit, I make 
connections to other 
subject areas. 


o 


o 


o 


o 


o 


o 


o 



28. Approximately how many artful Learning Units have you planned this school year? 



None 


1-2 


3-4 


5-6 


More than 6 


O 


O 


O 


O 


O 



29. In which of the following subject areas have you incorporated an Artful Learning Unit? 



Math 


Science 


Social Studies 


Language 


Other 


O 


O 


O 


O 


O 



30. What best describes your use of student portfolios for student assessment: 



a. I do not use student portfolios. 


O 


b. I use student portfolios, but I was already using them when I 
implemented the Artful Learning program in my classes. 


O 


c. I started using student portfolios because of my implementation of the 
Artful Learning program. 


o 
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3 1 . Based on the following scale: 





Never 


Once 


2-3 times 


4-5 times 


5 times or 
more 


Not 

applicable 
(students’ 
don’t have 
portfolios) 


a. How often during an Artful Learning Unit 
do you have students add to their portfolios? 


O 


O 


O 


O 


o 


O 


b. How often during an Artful Learning Unit 
do you have students reflect on the work in 
their portfolios? 


O 


o 


O 


O 


o 


O 


c. How often do you as a teacher assess your 
instruction of an Artful Learning Unit? 


o 


o 


o 


o 


o 


o 



32. Which kinds of technology have you incorporated into your Artful Learning Unit activities? 



Audio 


Video 


Computers/Internet 


Musical 


O 


O 


O 


O 



33. Based on the following scale: How often do you typically incorporate technology into an Artful Learning Unit? 



Daily 


At least twice 
a week 


Weekly 


Bi-weekly 


Monthly 


O 


O 


O 


O 


O 



34. Based on a typical Artful Learning Unit you’ve implemented, approximately what percentage of time is spent on the 



following components of the Artful Learning Model? 





0% 


25% 


50% 


75% 


100% 


a. 


Experience 


O 


O 


o 


o 


o 


b. 


Inquire 


O 


o 


o 


o 


o 


c. 


Create 


o 


o 


o 


o 


o 


d. 


Reflect 


o 


o 


o 


o 


o 



35. Based on the following scale: 





Never 


Rarely 


Sometimes 


Often 


Very often 


Always 


a. When planning an Artful Learning Unit, 
how often do you explicitly identify the 
content standards you are addressing? 


O 


O 


O 


O 


O 


O 


b. When planning an Artful Learning Unit, 
how often do you explicitly identify the 
visual, kinesthetic, or auditory 
experiences your students will engage in? 


O 


O 


O 


o 


O 


o 


c. How often when planning an Artful 

Leaning Unit, do you explicitly identify 
what higher level thinking skills your 
students will be engaged in? 


o 


O 


o 


o 


o 


o 
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36. To what extent do you agree or disagree with the following statements? 





Strongly 

disagree 


Disagree 


Disagree 
slightly 
more than 
agree 


Agree 
slightly 
more than 
disagree 


Agree 


Strongly 

agree 


Not 

applicable 


a. I understand the role of a 
significant question in the 
Artful Learning process. 


O 


O 


O 


O 


O 


o 


O 


b. When planning an Artful 

Learning Unit, I allow time for 
reflection at the end of the unit. 


O 


O 


O 


O 


O 


o 


O 


c. I give students opportunities to 
share with each other what 
they’ve learned during an 
Artful Learning Unit. 


o 


o 


o 


o 


o 


o 


o 



37. Did your school have an Artful Learning coach to facilitate your implementation of the program? 



O Yes 



O No 



O Don’t know 



38. If you answered yes to #37, how useful were the following types of support the coach provided: 





Not 
at all 
useful 


Slightly 

useful 


Somewhat 
useful - 
would have 
liked more 
support in 
this area 


Useful - 
I received 
support in 
this area 


Very useful - 
I received 
more than 
adequate 
support in this 
area 


N/A- 
our coach 
did not 
provide this 
type of 
support. 


a. 


Group professional development 
seminars 


O 


O 


O 


O 


O 


O 


b. 


Individual coaching on classroom 
practices 


O 


o 


O 


O 


O 


O 


c. 


Assistance in developing specific 
Artful Learning Units 


o 


o 


o 


o 


o 


o 


d. 


Assistance in analyzing/using 
assessment data 


o 


o 


o 


o 


o 


o 


e. 


Team building/support for team 
meetings and other teacher collaboration 


o 


o 


o 


o 


o 


o 



39. Based on the following scale: 





Never 


Rarely 


Sometimes 


Often 


Very often 


Always 


a. How often do you incorporate the creation of 
an artistic work (e.g. musical piece, drama, 
painting, poem. . .) into your instruction? 


O 


O 


O 


O 


O 


O 


b. How often do you meet with arts or 
music specialists to discuss your Artful 
Learning unit? 


O 


O 


O 


o 


O 


o 


c. How often do you share you own 
creative works with your students? 


o 


o 


o 


o 


o 


o 
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40. Comparing the implementation of the Artful Learning program at your school this year to previous years, do you feel that: 





Decreased 


Stayed the same 


Increased 


a. 


The number of teachers implementing the program has: 


O 


O 


O 


b. 


The amount of collaboration among teachers has: 


O 


O 


o 


c. 


Your administrators’ support for the program has: 


o 


o 


o 


d. 


The amount of support your school receives from the Grammy 
Foundation has: 


o 


o 


o 


e. 


Your use of the program in your classroom has: 


o 


o 


o 


f 


Your use of the Artful Learning portfolio assessments has: 


o 


o 


o 


g- 


Your understanding of the Artful Learning program components has: 


o 


o 


o 



4 1 . What impact do you feel the Artful Learning program has had on: 





Don’t 

know 


Negative 

impact 


No 

impact 


Slightly 

increased 


Moderately 

increased 


Greatly 

increased 


a. Your students’ engagement/interest with 
the curriculum 


O 


O 


O 


O 


O 


O 


b. The quality of your students’ work 
(classroom assignments, homework, 
projects) 


O 


O 


O 


O 


O 


O 


c. Students’ scores on class tests of quizzes 


o 


o 


o 


o 


o 


o 


d. Students’ scores on standardized tests 
(e.g., district or statewide assessments) 


o 


o 


o 


o 


o 


o 



42. This question is designed to help us gain a better understanding of teachers’ opinions of various school conditions, 
situations, and activities. Please indicate your opinion about each of the statements below. 



Your answers will be kept strictly confidential. 





Strongly 

agree 


Moderately 

agree 


Agree 
slightly 
more than 
disagree 


Disagree 
slightly 
more than 
agree 


Moderately 

disagree 


Strongly 

disagree 


a. If a child doesn’t learn something the first 
time, teachers will try another way. 


o 


o 


O 


O 


O 


O 


b. Teachers in this school are skilled in various 
methods of teaching. 


o 


o 


O 


O 


O 


O 


c . Teachers here are well-prepared to teach the 
subjects they are assigned to teach. 


o 


o 


o 


o 


o 


o 



(continues on next page) 
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(Continued) 





Strongly 

agree 


Moderately 

agree 


Agree 
slightly 
more than 
disagree 


Disagree 
slightly 
more than 
agree 


Moderately 

disagree 


Strongly 

disagree 


d. 


Teachers in this school really believe every 
child can learn. 


o 


o 


O 


O 


O 


O 


e. 


If a child doesn’t want to learn, teachers here 
give up. 


o 


o 


O 


O 


O 


o 


f. 


Teachers here fail to reach some students 
because of poor teaching methods. 


o 


o 


o 


o 


o 


o 


g- 


Teachers here don’t have the skills needed to 
produce meaningful student learning. 


o 


o 


o 


o 


o 


o 


h. 


Teachers in this school have what it takes to 
get the children to learn. 


o 


o 


o 


o 


o 


o 


i. 


Teachers in this school are able to get through 
to difficult students. 


o 


o 


o 


o 


o 


o 


j- 


Teachers here are confident they will be able 
to motivate their students. 


o 


o 


o 


o 


o 


o 


k. 


The lack of instructional materials and 
supplies makes teaching very difficult. 


o 


o 


o 


o 


o 


o 


1. 


Teachers in this school do not have the skills 
to deal with student disciplinary problems. 


o 


o 


o 


o 


o 


o 


m. 


Teachers in this school think there are some 
students that no one can reach. 


o 


o 


o 


o 


o 


o 


n. 


The quality of school facilities here really 
facilitates the teaching and learning process. 


o 


o 


o 


o 


o 


o 


0. 


Home life provides so many advantages that 
students are bound to learn. 


o 


o 


o 


o 


o 


o 


P- 


These students come to school ready to learn. 


o 


o 


o 


o 


o 


o 


q- 


Drugs and alcohol abuse in the community 
make learning difficult for students here. 


o 


o 


o 


o 


o 


o 


r. 


The opportunities in this community help 
ensure that these students will learn. 


o 


o 


o 


o 


o 


o 


s. 


Students here just aren’t motivated to learn. 


o 


o 


o 


o 


o 


o 


t. 


Learning is more difficult at this school 
because students are worried about their 
safety. 


o 


o 


o 


o 


o 


o 


u. 


Teachers here need more training to know 
how to deal with these students. 


o 


o 


o 


o 


o 


o 



Thank you for your participation in this survey! 
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